PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldMark: A Unified Benchmark Suite for Interactive Video World Models

Xiaojie Xu, Zhengyuan Lin, Kang He, Yukang Feng, Xiaofeng Mao, Yuanyang Yin, Yongtao Ge, Kaipeng Zhang

arXiv:2604.21686v2cs.CV

TL;DR

Interactive world-model evaluation lacks a shared protocol across incompatible controls and largely reduces action following to coarse trajectory or direction errors. WorldMark uses per-model adapters and a shared action vocabulary across 500 standardized cases, then evaluates action dynamics per axis alongside memory and visual quality. It reveals a response–stability trade-off that existing single metrics cannot capture.

  • Problem

    Existing benchmarks serve visual quality and world memory but do not fairly compare incompatible control interfaces or resolve action responses beyond trajectory or direction error.

  • Method

    WorldMark translates six shared WASD+L/R primitives through per-model adapters and evaluates ten models on 500 standardized cases with per-axis action-dynamics, memory, and visual-quality metrics.

  • Results

    The benchmark exposes a response–stability trade-off: models with similar direction accuracy can differ substantially in latency and stability, and the fastest responders are often the least stable.

  • Takeaways & Limitations

    Action following requires measuring when motion begins, whether it stays on the commanded axis, and whether it persists, rather than relying on one aggregate trajectory score.

  • Takeaways & Limitations

    WorldMark omits object dynamics within scenes and event-based interaction.

Abstract

from arXiv · show

Unlike text- or image-driven video generation, an interactive world model is driven by actions: the user acts, and the world responds. Two obstacles stand in the way of fair and comprehensive evaluation. First, models take actions in incompatible formats---captions, camera trajectories, action functions---so no shared protocol has been established. Second, while existing benchmarks have advanced world memory and visual quality, action following is reduced to trajectory or direction error, which collapses a whole path into one number: not how quickly the world reacts to a command switch, nor how cleanly it moves along the commanded axis. WorldMark removes both obstacles. Per-model adapters translate a shared WASD-style vocabulary into each model's native control format, so ten heterogeneous models receive semantically identical instructions across 500 standardized cases spanning styles, viewpoints, and difficulty tiers; a new model costs one adapter. On this common ground we characterize action dynamics through a control-systems lens---direction accuracy, direction purity, response latency, and motion stability, each resolved per axis---alongside suites for world memory and visual quality. Together they expose differences existing protocols cannot see: the fastest responders are often the least stable, a trade-off no single action metric captures; per-axis resolution reveals models that follow translation almost perfectly while barely responding to rotation; the model with the best perceptual and aesthetic quality ranks last in translational direction accuracy and latency; and stylized scenes cost every model global consistency while leaving action dynamics largely intact. We will release all data, evaluation code, and model outputs.

1 Introduction

WorldMark addresses incompatible action interfaces and overly coarse action-following evaluation with a unified benchmark and control-systems-based metrics. It evaluates ten models on standardized cases while also measuring world memory and visual quality.

  • Motivation: Existing benchmarks emphasize visual quality and memory, while action following is constrained by incompatible interfaces and trajectory or direction errors that collapse response behavior.These coarse measures do not reveal off-axis drift, delayed reactions after command switches, or stalling.
  • Unified benchmark: WorldMark translates six shared WASD+L/R primitives through per-model adapters, enabling semantically identical instructions for ten heterogeneous models.Adding a model requires one adapter.
  • Unified benchmark: The benchmark covers 500 standardized cases spanning real and stylized scenes, first- and third-person views, and three difficulty tiers.The cases use 50 scenes and 15 action trajectories.
  • Metrics: Four per-axis Action Dynamics metrics measure direction accuracy, direction purity, response latency, and motion stability, alongside memory and visual-quality metrics.The action protocol treats commands as step inputs and evaluates how motion begins, follows the commanded axis, and persists.

2 Related Works

Prior benchmarks cover visual quality, physical plausibility, camera-conditioned consistency, or world memory, but differ in interfaces and often lack common interactive cases. WorldMark unifies action evaluation across models while adding response dynamics and an integrated revisit probe.

  • Prior benchmarks: Early interactive video world models established action-conditioned generation in Minecraft, but that setting was confined to a single environment.
  • Prior benchmarks: Current interactive models use heterogeneous controls including caption keywords, 6-DoF poses, camera trajectories, and discrete action functions.
  • Prior benchmarks: General-purpose and camera-conditioned benchmarks measure generation quality, physical plausibility, or reconstructed geometry without placing the user in the loop.
  • Prior benchmarks: Concurrent interactive benchmarks handle heterogeneity through ground-truth poses, separate text and action tracks, or other restrictions that limit direct score comparability.
  • WorldMark: WorldMark uses per-model adapters for identical action sequences, video-only metrics across models, per-axis response dynamics, and a revisit probe based on equal accumulated motion.

3 WorldMark Suite

WorldMark combines standardized image and action suites with a unified interface and nine evaluation dimensions spanning action dynamics, world memory, and visual quality. Its metrics use optical flow and scale-free comparisons to distinguish response direction, latency, stability, persistence, memory failures, and appearance quality.

  • Suite components: WorldMark comprises nine metrics across Action Dynamics, World Memory, and Visual Quality, alongside standardized images, trajectories, and a reproducible workflow.Users can also apply custom metrics to the standardized image–action pairs.
  • Action Dynamics: Action Dynamics separately measures commanded-direction accuracy, off-axis purity, response latency, and motion stability for each motion type.Latency measures time to reach half the model’s normally sustained speed, while stability penalizes inactivity, intermittent motion, and terminal slowing.
  • Action Dynamics: Optical flow replaces camera-pose reconstruction, and depth-based parallax separates translation from rotation while keeping motion comparisons scale-free across models.The method decomposes horizontal flow using inverse-depth variation and fits the components by least squares.
  • World Memory and Visual Quality: World memory is evaluated at local, global, and revisit timescales, while frame quality tests whether action dynamics and memory reduce to appearance.Local memory detects cuts and adjacent-frame mutations; global memory measures geometric drift; revisit memory pairs outbound and return frames by accumulated flow.
  • Image and action coverage: The suite covers diverse scenes, styles, viewpoints, and 15 action trajectories, with near-duplicate reduction preserving diversity at a fortieth of the original pool.The image suite uses 25 photorealistic and 25 stylized scenes, paired across first- and third-person views.
  • Unified Action Interface: A per-model adapter translates six WASD+L/R primitives into keyboard, caption, or camera-control formats while retaining each model’s native step size and yaw rate.This avoids imposing assumptions on otherwise incommensurable motion units and accommodates heterogeneous interfaces.

4 Experiments

WorldMark evaluates ten heterogeneous interactive world models across standardized scenes, viewpoints, and action sequences, revealing action, memory, viewpoint, style, and quality patterns that aggregate direction scores miss.

  • Action dynamics: Eight of ten models exceed 88 in translational Direction Accuracy, while Motion Stability spans 31.7–87.2 for translation and 13.7–98.1 for rotation.Translational Latency also spans 40.7–98.2, showing substantial differences among models with similar direction scores.
  • Action dynamics: DreamX-World reaches Response Latency scores of 96.1 translation and 97.0 rotation but has rotational Stability of 27.7, revealing a roughly 50-point response–stability gap.HY-World 1.5 and SANA-WM show similar gaps of approximately 26 and 22 points.
  • Action dynamics: HY-GameCraft 1.0 follows translation with Direction Accuracy 89.1 but barely responds to rotation at 2.7, while AlayaWorld performs better on rotation than translation.HY-GameCraft’s rotational Stability of 13.7 and Latency of 52.0 indicate an inoperative rotational axis rather than mere imprecision.
  • World memory: Local Memory is concentrated near the best score, whereas Global Memory ranges from 34.0–75.8; DreamX-World combines smooth local motion of 93.3 with global coherence of 34.0.The three timescales prevent a saturated local component from concealing long-range inconsistency.
  • Viewpoint and style: Third-person views reduce rotational Direction Accuracy by 13.9 points and rotational Purity by 12.2 on average, compared with translational losses of 4.0 and 2.4.The losses are uneven: Matrix-Game 2.0 and LingBot-World are most affected, while SANA-WM and DreamX-World are unaffected.
  • Viewpoint and style: Stylization lowers Local Memory by 7.9 points and Global Memory by 7.4 on average across all ten models, while translational Direction Accuracy changes by only −1.5.This separates style-related consistency degradation from largely stable command following.
  • Visual quality: Perceptual Quality correlates negatively with translational action dynamics, most strongly with Response Latency at ρ = −0.82, despite positive correlations with memory metrics.Yume 1.5 ranks first in Perceptual Quality at 87.2 but last in translational Direction Accuracy at 51.9 and Latency at 40.7.
  • Metric validation: Metric–human agreement averages 85.1% across ten dimensions, ranging from 79.1% for rotational Stability to 93.3% for Revisit Memory.The study used 40 annotators, each comparing 20 blinded pairs per metric; Global Memory and Visual Quality were excluded for stated reasons.

5 Conclusion and Limitations

WorldMark unifies action evaluation for ten interactive I2V world models over 500 cases using adapters and per-axis step-response metrics, exposing latency–stability and quality–responsiveness differences. The benchmark intentionally excludes object dynamics and event-based interaction.

  • Conclusion: WorldMark drives ten control formats with identical actions over 500 cases and scores direction, purity, latency, and stability per axis alongside memory and visual quality.The nine deterministic metrics treat action following as a step response.
  • Conclusion: Models with similar direction accuracy can differ several-fold in latency, while the fastest responders are often the least stable and the best-looking model is among the least responsive.These findings demonstrate why action dynamics cannot be represented by a single direction score or visual-quality ranking.
  • Limitations: WorldMark does not score object dynamics within scenes or event-based interaction.Object dynamics are left to existing benchmarks, while event-based interaction is described as largely confined to text-to-video models at present.

Supplementary Materials for WorldMark

The supplementary materials document WorldMark’s scene selection, coverage, paired viewpoints, action-sequence construction, captioning, third-person synthesis, and adapter pipeline across heterogeneous native interfaces.

  • A. Image Suite: The Image Suite draws from WorldScore’s filtered pool of 1000 photorealistic and 1000 stylized scenes across ten subcategories per domain.WorldMark further removes scenes too enclosed or uniform to support meaningful 20–60 second navigation.
  • A. Image Suite: Figure 6 shows the retained references scattered across the occupied CLIP feature-space region in all eight t-SNE panels.The retained set therefore preserves the pool’s coverage rather than selecting only a narrow visual cluster.
  • A. Image Suite: The 100 images form 25 paired scenes per domain, with first-person references matched to third-person composites of the same frame.The controlled pair differs only around the added character, and five sequences per image yield 500 evaluation cases.
  • B. Action Suite: References are captioned separately for the scene and main character before Gemini-3.1-Pro selects five suitable sequences from a 15-sequence action library.The action vocabulary contains W/A/S/D translation, L/R viewpoint rotation, and sequences lasting 20 seconds per action.
  • B. Viewpoint construction: Third-person counterparts are synthesized by adding a fully visible rear-view character in the lower middle while preserving the original resolution and background except where placement conflicts.This creates a same-camera comparison rather than a second viewpoint.
  • C. Unified interface: The adapter parses each benchmark sequence into translation, yaw, and duration triples before mapping it into each model’s native control signal.The intermediate representation prevents repository-specific meanings of the same key from breaking semantic alignment.
  • C. Unified interface: Table 5 records native interface paths and model-specific translation and yaw scales, which are not calibrated to a common magnitude.The suite preserves semantic identity while allowing each model to operate through its own native interface.

C.1 Per-Model Adaptation

Per-model adapters accommodate distinct control representations, timing granularities, camera conventions, and native motion scales; evaluation preserves realized segment boundaries and tests whether commands arrive correctly.

  • Native adapters: AlayaWorld consumes camera-to-world matrices with image, prompt, and intrinsic files; a 20-second segment contains 480 frames across fifteen 32-frame chunks.Its adapter initializes the first camera matrix as the identity.
  • Native adapters: Native interfaces vary in action semantics and timing: HY-World uses duration-bearing pose strings, HY-GameCraft repeats fixed 33-frame actions, and LingBot-World requires frame counts satisfying 4n+1.Matrix-Game models use keyboard or mouse vectors, while SANA-WM applies smoothed target speeds with rise and coast dynamics.
  • Native adapters: Lyra 2.0 converts integrated camera-to-world motion into tiled world-to-camera trajectories, while Matrix-Game 3.0 uses per-frame keyboard and mouse signals with model-specific chunking.These adapters preserve each model’s required representation rather than forcing a shared geometric format.
  • Native adapters: Yume 1.5 represents each 20-second segment with ten two-second caption lines, separating movement and camera slots so in-place rotation is expressible.The fixed template can state that the person stands still while the camera turns.
  • Scale and timing: A common 20-second right turn spans about 192° in SANA-WM, 480° in Matrix-Game 3.0, 640° in LingBot-World, and 1440° in HY-World 1.5.Therefore semantic identity is defined over action category and temporal structure, not magnitude.
  • Scale and timing: Segments are scored using their realized frame boundaries without resampling or cropping, so each metric normalizes within the segment actually produced by the model.This avoids changing apparent speed when native rollout granularities differ.
  • Adapter validation: Easy-tier translational Direction Accuracy averages 88.2 on Real and 87.2 on Stylized, while rotation averages 77.6 overall and 84.5 excluding inoperative HY-GameCraft rotation.The adapter equalizes direction but not native motion magnitude.
  • Metric calibration: Table 7 distinguishes thresholds that affect reported scores from structural preparation settings, while Direction Accuracy and Direction Purity use ratios without scalar thresholds.Only τmut is identified as the genuine decision threshold among the calibrated constants.

D.2 Ranking Stability Under Perturbation

WorldMark’s leaderboard is largely stable under one-at-a-time perturbations of its scoring thresholds, with the main exceptions arising from Motion Stability time constants.

  • Threshold sweep: Fifteen of 17 scoring thresholds keep the leaderboard within one place on both splits at ρ ≥0.976, while ten preserve the ordering bit-identically.The sweep re-scores cached intermediates and validates the default scorer on every run.
  • Exceptions: Stretching the stall debounce from 0.3 s to 0.5 s lifts DreamX-World from eighth to fifth (ρ = 0.927).The longer debounce erases many of DreamX-World’s stalls shorter than half a second.
  • Constant taxonomy: Of 46 pipeline constants, 17 are scoring thresholds and 27 are structural settings; Direction Accuracy and Direction Purity have no scalar threshold.All constants are fixed across models, splits, and difficulty tiers, with provenance recorded separately.
  • Exceptions: Stretching the velocity smoothing window moves a model two places on Stylized (ρ = 0.964).Both ranking exceptions involve Motion Stability time constants, while the defaults remain inside a stable regime.
  • Metric coverage: Response Latency and Revisit Memory are undefined on Easy, while other metric availability depends on the trajectory and exercised motion axes.Response Latency requires a command switch, and Revisit Memory requires an equal-duration primitive–inverse pair.

F.1 Why We Use Optical Flow Rather Than Pose

WorldMark uses optical flow because per-frame response metrics require stable velocity estimates, whereas differenced pose streams amplify noise and can fail on particular clips.

  • Signal choice: Optical flow directly measures per-frame velocity, while pose estimates reconstruct position and require differencing whose noise scales as 1/∆t.This distinction matters because Response Latency and Motion Stability depend on frame-level sampling.
  • Noise comparison: Median per-frame velocity noise is 0.08 on Real and 0.07 on Stylized for flow, versus 0.28 and 0.27 for SLAM and 0.63 and 0.62 for the feed-forward estimator.Flow is both the lowest-noise and tightest signal across domains.
  • Failure modes: On roughly one clip in eight, SLAM noise exceeds flow by two to three orders of magnitude, peaking at 1119 when tracking is lost.Outside those failures, SLAM is near 3.5 times flow’s median, while the feed-forward estimator is near eight times flow’s median.
  • Failure modes: Flow recovers the commanded reversal on every inspected clip, while feed-forward pose jitters around zero on five of six clips and SLAM fails when tracking is lost.SLAM failures include flat traces with spikes and large noise excursions; feed-forward failure is tied to differenced pose error dominating motion.
  • Evaluation scope: The six frame-by-frame failure examples were selected by ranking for low flow noise and high pose noise, so they exhibit rather than randomly sample the failure.Depth Anything 3 remains in the pipeline for depth estimation, not as a motion source.

G.3 Per-Metric Agreement

Human judgments generally agree with WorldMark’s per-metric rankings, though agreement varies by dimension, with Direction Purity weaker and ties more common in rotation.

  • Overall agreement: 85.1% average agreement across ten dimensions ranges from 79.1% for rotational Motion Stability to 93.3% for Revisit Memory.Agreement is the fraction of expressed preferences matching the metric, averaged across forty annotators.
  • Dimension differences: Direction Purity is the weakest dimension, with agreement of 79.4 and 83.2 across its two axes and the highest tie rates, 14.8% and 14.9%.The negative agreement–tie relationship is r = −0.76 (p = 0.011).
  • Axis differences: Translation reaches 85.1% agreement versus 83.3% for rotation, while tie rates are 9.6% and 13.4%, respectively.Axis differences are significant for tie rates on Direction Accuracy and Motion Stability, and for Motion Stability agreement.
  • Reporting limitation: The study records per-annotator totals but not pair-level responses, preventing computation of Fleiss’ κ.This limits the available inter-rater reliability analysis.

H Quantitative Results by Difficulty Tier

Increasing difficulty mainly damages long-range geometry and rotational control, while local memory, perceptual quality, and translational accuracy change little.

  • Metric availability: Response Latency and Revisit Memory are undefined on Easy and excluded from its ranking because a single-key action neither switches command nor returns.They are marked unavailable rather than scored as zero.
  • Memory and quality: Global Memory falls 35.0% from Easy to Hard on Real, from 62.3 to 40.5, and declines for all ten models.The corresponding Stylized decline is 40.7%.
  • Memory and quality: Perceptual Quality declines 8.6%, Local Memory declines 5.4%, and Aesthetic Quality remains flat at 0.9% across difficulty tiers.Difficulty therefore affects long-range geometry more than these other measures.
  • Translation: No model reaches 90 on translational Direction Purity in any tier on either split, with ceilings of 86.4 on Real and 82.5 on Stylized.The median Accuracy-minus-Purity gap remains larger for translation than rotation at every tier.
  • Rotation: Rotational Direction Accuracy falls from Easy to Hard for eight of ten models on Real, while translational Accuracy changes from 88.2 to 88.0.This identifies rotation, rather than translation, as the action dimension most affected by difficulty.

I Per-Model Capability Profiles

Lyra 2.0 is the sole broadly capable model, while stylization penalizes the strongest Real-split models most heavily.

  • Lyra 2.0 is the only near-convex profile, meeting or exceeding the suite mean on 12 of 13 columns.HY-World 1.5 reaches 10 columns, and no other model exceeds 9.
  • Every remaining profile has at least four notches, indicating no broad group of near-equal generalists.
  • 38.8 to 110.6 is the stylization-penalty range across models, with Lyra 2.0 and AlayaWorld highest at 110.6 and 98.3.DreamX-World has the smallest summed penalty, while Lyra 2.0 and AlayaWorld lead on Real.
  • Response Latency and Revisit Memory are undefined on Easy and excluded from the Easy ranking.

J Qualitative Examples per Metric

The qualitative examples show how each metric isolates a distinct failure mode, pairing fixed scenes and commands with motion traces or video frames.

  • Each metric comparison holds the scene and command fixed while contrasting one high- and one low-scoring model.Action Dynamics examples include a signal trace and six sampled frames; whole-video metrics use the frames their metric reads.
  • Direction Accuracy measures whether commanded-axis flow has the correct sign, while Direction Purity measures the commanded axis’s share of total motion.Purity uses an unsigned energy split across three channels, so it measures axis allocation rather than direction.
  • Direction Accuracy reaches 0 when a forward-then-backward sequence never crosses the commanded axis.
  • Direction Purity falls when turning produces more strafing than yaw, leaving the commanded channel smaller than another motion channel.
  • Motion Stability reaches 0 for oscillating, decaying motion, while Response Latency distinguishes an immediate step from a seconds-late ramp.
  • Local Memory reaches 0 when a sequence abruptly cuts from an open mountainside to a building interior mid-command.
Loading 2604.21686v2…