PaperlayerView exact revision on arXiv

Source-linked AI summary

iWorld-Bench: A Benchmark for Interactive World Models with a Unified Action Generation Framework

Jianjie Fang, Yingshan Lei, Qin Wan, Ziyou Wang, Yuchao Huang, Yongyan Xu, Baining Zhao, Weichen Zhang, Chen Gao, Xinlei Chen, Yong Li

arXiv:2605.03941v2cs.CVcs.AI

TL;DR

Interactive world models lack a dedicated framework that tests responsiveness to actions, varied task difficulty, and memory. iWorld-Bench unifies evaluation across six interactive task types, and its evaluation of 14 models reveals interaction limitations and trade-offs between generation quality and controllability.

  • Problem

    Existing evaluations lack tasks that test interactive world models’ responsiveness to external action sequences, varying difficulty, and memory.

    Source: §1 · p. 2

  • Method

    iWorld-Bench uses an Action Generation Framework to unify evaluation across modalities through six task types and 4,900 evaluation tasks.

    Source: §1 · p. 3

  • Results

    Evaluations of 14 interactive world models reveal interaction limitations and trade-offs between generation quality and controllability.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The evaluation highlights differences and shortcomings across world-model paradigms in interaction capability.

    Source: §5 · p. 9

Abstract

from arXiv · show

Achieving Artificial General Intelligence (AGI) requires agents that learn and interact adaptively, with interactive world models providing scalable environments for perception, reasoning, and action. Yet current research still lacks large-scale datasets and unified benchmarks to evaluate their physical interaction capabilities. To address this, we propose iWorld-Bench, a comprehensive benchmark for training and testing world models on interaction-related abilities such as distance perception and memory. We construct a diverse dataset with 330k video clips and select 2.1k high-quality samples covering varied perspectives, weather, and scenes. As existing world models differ in interaction modalities, we introduce an Action Generation Framework to unify evaluation and design six task types, generating 4.9k test samples. These tasks jointly assess model performance across visual generation, trajectory following, and memory. Evaluating 14 representative world models, we identify key limitations and provide insights for future research. The iWorld-Bench model leaderboard is publicly available at iWorld-Bench.com.

1. Introduction

iWorld-Bench is a benchmark for evaluating interactive world models across diverse scenes, action modalities, and interaction abilities. It combines a standardized dataset and action-generation framework with varied tasks and metrics to assess models’ visual quality, action following, and memory.

1. Introduction

  • Existing benchmarks lack diverse scenes and perspectives, unified action inputs, and tasks that test responsiveness to external actions and memory.Heterogeneous action representations make fair comparisons difficult, while existing task designs often neglect varying difficulty levels.Source: §1 · pp. 1–2
  • iWorld-Bench builds on 330,000 high-quality clips and selects 2,100 videos for evaluation, covering four observation perspectives, varied weather and lighting, and diverse scenes.Its processing protocol standardizes 12 datasets’ coordinate systems and camera parameter formats; its collection pipeline adds clips from 18 environments across four simulators.Source: §1 · p. 2
  • A modality-agnostic Action Generation Framework represents 81 fundamental motion actions across modalities, enabling unified evaluation of different world models.The framework defines an action-space dictionary and supports additional modality encodings.Source: §1 · pp. 2–3
  • Six trajectory-based task types assess interaction across difficulty levels, including memory tests and diverse interactive actions.The benchmark’s tasks vary in trajectory count and are designed to evaluate models in dynamic, complex scenarios.Source: §1 · p. 2
  • The benchmark evaluates 14 interactive world models using 4,900 tasks and 9 metrics spanning visual quality, action following, and memory.The evaluated models include text-controlled, one-hot encoding-based, and camera-parameter-controlled systems.Source: §1 · p. 3

2. Related Work

Related work spans camera-annotated datasets, interactive-world-model benchmarks, and interaction modalities. iWorld-Bench draws on diverse dataset sources and introduces a unified framework to evaluate interactions across models.

Datasets with Camera Parameters

  • Prior datasets span autonomous driving, robotics, and drone data; 3D reconstruction collections; and large-scale world-model datasets, informing iWorld-Bench’s diverse, all-weather curation.The cited sources provide dynamic scenes, varied indoor and outdoor environments, and camera parameters.Source: §2 · p. 3

World Model Benchmark

Interactive World Model

3. iWorld-Bench

iWorld-Bench unifies evaluation of world-model interaction across modalities, using a 330,000-clip corpus, 2,100 selected videos, 4,900 tasks, six task types, and nine metrics.

3.1. Dataset Pipline

  • The dataset pipeline combines 12 camera-parameterized datasets with data from 18 scenes across four simulators, standardizes the sources, and selects 2,100 videos from 330,000 clips.The benchmark overview describes coverage of five indoor lighting conditions, nine outdoor weather types, thousands of scenes, and tens of millions of entities.Source: §3 · p. 4, §3.1 · p. 4, §3.1.1 · p. 4

3.1.2. VIDEO GENERATE CREATE FUTURE

  • Outdoor expansion collected 100,000 videos from 18 scenes across four simulators using 450 points and staged frame- and sample-level filtering; VLM annotations were checked by multiple models and refined by humans.The unified annotations cover environment, scene, weather or lighting, and entities; representative videos were selected for coverage across scene categories, weather, and entity types.Source: §3.1.2 · p. 4, §3.1.3 · p. 5

3.2.1. ACTION GENERATION FRAMEWORK

  • The action framework defines a 729-action motion space and maps an 81-action commonly supported subset to camera parameters, one-hot codes, and text controls across modalities.Actions are encoded through difficulty, translation, rotation, and validity, allowing compatible control signals and extensible task design.Source: §3.2 · p. 5, §3.2.1 · p. 6

3.2.2. IWORLD-BENCH DESIGN

  • Four action-control tiers test 9, 24, 32, and 16 actions at difficulties 1–4, respectively, with 1,000 tasks per tier.The tiers progress from basic actions to four-degree-of-freedom complex tasks.Source: §3.2.2 · p. 6
  • Memory uses 200 cyclic-path tasks to assess generation consistency and trajectory alignment; camera following uses 700 parameter files to test trajectory control.Memory tasks require returning to the same location during one inference, while camera following targets models controlled by intrinsic and extrinsic parameters.Source: §3.2.2 · pp. 6–7

3.3.1. GENERATION QUALITY

  • Generation-quality metrics assess image fidelity with normalized MUSIQ, brightness consistency, HSV hue stability, and sharpness using Tenengrad with a BRISQUE-triggered artifact safeguard.The metrics target low-level visual distortion, temporal brightness shifts, hue drift, and noise-induced or high-frequency sharpness artifacts.Source: §3.3.1 · p. 7

3.3.2. TRAJECTORY FOLLOWING

  • Trajectory metrics assess motion smoothness with LPIPS, SSIM, and MSE, ViPE-based directional accuracy, and ground-truth-calibrated trajectory tolerance to reduce estimator variance.Together, these measures target unnatural jitter, instruction-level controllability, and physical execution fidelity.Source: §3.3.2 · p. 7

3.3.3. MEMORY ABILITY

  • Memory metrics measure pixel consistency of symmetric loop frames and mirror similarity of displacement vectors to assess loop closure and spatial persistence.They target memory decay or structural logic failure during extended temporal reasoning and reciprocal return trips.Source: §3.3.3 · p. 7

4. Experiments

Experiments evaluate 14 representative interactive world models across human-validated metrics, Action Control and Memory Ability tasks, and Camera Following. Results show different strengths across control paradigms and a trade-off between visual fidelity and action controllability.

4.1. Setup

  • The evaluation covers 14 representative models spanning text-conditioned, one-hot–conditioned, and explicit camera-parameter control, with inference run on NVIDIA A800 GPUs.Source: §4.1 · p. 7

4.2.1. HUMAN PREFERENCE VALIDATION

  • Human preference results show that iWorld-Bench metrics align well with subjective perceptions and remain robust to changes in video resolution and aspect ratio.Source: §4.2.1 · p. 7

4.2.2. ACTION CONTROL AND MEMORY ABILITY

  • One-hot models show stronger interaction capabilities but less flexibility than camera-parameter control, while text-controlled models favor generation quality over trajectory following.CogVideoX-I2V has the highest Brightness Consistency (0.8988), but text-controlled models generally have weaker trajectory following; HY-World 1.5 and VideoX-Fun-WAN show balanced profiles.Source: §4.2.2 · pp. 7–8, Figure 3 · p. 9
  • HY-World 1.5 ranks first overall at 0.7873, with 0.7472 trajectory accuracy versus CogVideoX-I2V’s 0.5950.The results attribute the stronger trajectory following to discrete action signals, while HY-World 1.5 also excels in memory ability.Source: §4.2.2 · p. 7, Table 3 · p. 8
  • Camera-control fine-tuning improves trajectory following but slightly degrades generation quality compared with base models.The passage gives AC3D fine-tuned from CogVideoX-I2V and HY-World 1.5 derived from HunyuanVideo-1.5 as examples.Source: §4.2.2 · p. 8

4.2.3. CAMERA FOLLOWING

  • AC3D leads camera-parameter-controlled models in Trajectory Tolerance (0.9091), Brightness Consistency (0.8927), and Motion Smoothness (0.9919).The results are based on 700 annotated camera files and attribute AC3D’s precise control to its use of explicit camera parameters.Source: §4.2.3 · p. 8, Table 4 · p. 8
  • Camera Following Trajectory Tolerance ranges from 0.4286 for ASTRA to 0.9091 for AC3D, while RealCam-I2V leads Image Quality at 0.5889 but has 0.7480 Trajectory Tolerance.The results describe visual fidelity and action controllability as distinct evaluation dimensions, with early methods CameraCtrl and MotionCtrl showing limited effectiveness.Source: §4.2.3 · p. 8, Table 4 · p. 8, Figure 3 · p. 9

5. Conclusion and Future Work

iWorld-Bench provides a unified, multidimensional benchmark and a 330,000-video dataset for evaluating interactive world models. Evaluations of 14 models reveal interaction limitations and trade-offs, while future work targets real-time performance and long-horizon consistency.

Conclusion and Future Work

  • iWorld-Bench evaluates interactive world models through six task types and assessments across diverse worlds, revealing limitations in current models’ interaction capabilities.The benchmark evaluates 14 interactive world models and their base models, analyzing paradigm differences and trade-offs between generation quality and controllability.Source: §5 · p. 9
  • A data processing pipeline produced a high-quality, multi-scene, multi-perspective dataset containing 330,000 videos for world model research.Source: §5 · p. 9
  • Future work will add evaluations of real-time performance and long-horizon consistency to make iWorld-Bench more comprehensive for the community.Source: §5 · p. 9

A. Dataset details … A.4. Data Annotation Prompts

The appendix describes the open-source datasets and simulators selected, a pipeline for filtering and standardizing heterogeneous video into 81-frame clips, and structured GPT-4o annotation of 330,000 videos.

A.1. Open Sources Datasets Details

  • Open-source inputs cover robot-simulation data, autonomous driving, spatially annotated video, and camera-pose, RGB-D, and SLAM datasets.Source: §A.1 · pp. 13–14

A.2. Simulator Details

  • The study selected 8 of aerial VLN’s 22 environments, 5 of UAV ON’s 15, all 4 Openfly environments, and Embodied City’s sole environment.Source: §A.2 · p. 14

A.3. Fliter

  • Frame-level visual anomalies are converted into local anomaly-density zones; clean zones use τ = 0.06, nearby segments are merged, and segments shorter than Lmin are discarded.The pipeline detects anomalies using brightness stability and chromatic mutation metrics.Source: §A.3 · pp. 17–18
  • The four-stage corpus pipeline acquires and synchronizes sources, standardizes them in CityWorld’s four-layer format with three convertible pose forms, rectifies coordinates, and packages synchronized 81-frame clips.Source: §A.3 · p. 18

A.4. Data Annotation Prompts

  • GPT-4o 2025-01-01-preview annotated 330,000 videos using 119 million input tokens and 21.86 million output tokens, at an estimated cost of approximately $518.Source: §A.4 · p. 18
  • The prompt analyzes five frames as valid JSON, requesting indoor/outdoor type, a scene description and root-noun tag, weather or lighting, and at least 15 visible entities.Source: §A.4 · p. 19

A.5. Multi-Model Verification and Human Refinement … B.3. Existing Keyboard and Mouse Support Space

The appendix describes a multi-model verification and human-refinement process for annotations, followed by a motion-action framework covering translation, rotation, and memory-oriented trajectories.

A.5. Multi-Model Verification and Human Refinement

  • Three independent VLMs verified all 330,000 clips; any disagreement triggered human review rather than majority-vote acceptance, with 10 volunteers spending ~1,200 hours and ~$2.8K in API fees.Human inspection covered disagreement cases and 10,000 unanimous-case spot checks, totaling 71,380 clips, or 21.6% of the corpus.Source: §A.5 · pp. 19–20
  • 98.8% of annotations required no modification; human review edited 6.35% of 61,380 flagged clips (~3,897, or 1.2% of the corpus).A spot-check of 10,000 unanimously accepted clips had a 100% pass rate.Source: §A.5 · p. 19

A.6. Human Annotation Procedures

  • Stratified sampling used VLM-generated spatial, atmospheric, and entity labels to diversify candidate videos, while an in-house tool supported first-frame verification, correction, task grading, and export.Annotators could designate Memory tasks and assign one of four difficulty levels based on visual complexity such as occlusion and background clutter.Source: §A.6 · p. 20

B. Framework Details

  • The framework proposes a unified, quantifiable, and extensible action space using a quadruple descriptor for motion translation and rotation.The descriptor is organized around Difficulty, ID, Direction, and Keys.Source: B. Framework Details · p. 21, §B.1 · p. 21

B.1. Action Mapping

  • Each of translation and rotation has 27 uniquely identified actions; the mapping encodes 3D position and camera orientation, with simultaneous parameter changes adding their difficulty values and stillness assigned difficulty 1.Actions without predefined keys are included so the framework can accommodate future key mappings.Source: §B.1 · p. 21, Table 7 · p. 21

B.2. Entire Action Space

  • The entire action space enumerates 729 translation–rotation combinations with difficulty levels 1–6; validity is based on action frequency in the collected dataset, with exceptional cases marked invalid.Complete stillness is assigned difficulty 1.Source: §B.2 · p. 21, Table 8 · p. 22

B.3. Existing Keyboard and Mouse Support Space

  • The keyboard-and-mouse support space contains 81 motions with difficulty levels 1–4, while the memory mapping defines inverses for six motion patterns to guide back-and-forth trajectory generation.The memory mapping includes cases without corresponding keys.Source: §B.3 · pp. 23–24, Table 9 · p. 23, Table 10 · p. 24

C. Benchmark Details · C.1. Data presentation of different difficulty

The benchmark’s Difference Verification task is illustrated across four difficulty levels, progressing from basic camera motions to complex multi-axis movement with view changes.

C.1. Data presentation of different difficulty

  • Figure 5 presents Difference Verification examples across four difficulty levels based on camera-operation complexity.Source: Figure 5 · p. 25
  • Row 1 shows basic single-axis camera movements.Source: Figure 5 · p. 25
  • Row 2 adds combined camera translation and rotation.Source: Figure 5 · p. 25
  • Row 3 features sequential composite camera trajectories.Source: Figure 5 · p. 25
  • Row 4 involves complex multi-axis movements with view changes.Source: Figure 5 · p. 25
  • The examples demonstrate model ability to detect subtle pose differences across scenarios.Source: Figure 5 · p. 25

C.2. Data presentation of memory

Figure 6 showcases the Memory Verification task’s loop-closure difficulty through reversible camera trajectories that require recalling the initial state. Red bounding boxes mark visual cues for temporal reasoning and consistency checks in long-term memory scenarios.

C.2. Data presentation of memory

  • Reversible camera actions, such as “up then down” or “turn right then left,” require recalling the initial state to verify loop closure.Source: Figure 6 · p. 26
  • Red bounding boxes highlight visual cues used for temporal reasoning and consistency checks in long-term memory scenarios.Source: Figure 6 · p. 26

D. Metrics details · E. Expriment details

The section defines nine metrics for video appearance and temporal consistency, motion smoothness, trajectory control, and memory-related symmetry and alignment. Together, they assess visual fidelity and coherence as well as whether generated motion follows commanded or symmetric routes.

D. Metrics details

  • Image Quality (SImage) scores frame-level visual fidelity with MUSIQ across the video, then averages and linearly normalizes the frame scores.It targets low-level distortions such as overexposure, noise, and blur.Source: D. Metrics details · p. 26
  • Brightness Consistency (SBrightness) compares each frame’s dark, mid, and bright pixel distribution with the initial frame, using similarity and normalized temporal decay weights.The three-level classification allows reasonable brightness changes while helping detect style collapse and brightness drift in long sequences.Source: D. Metrics details · pp. 26–27
  • Color Temperature Constraint (SColor) compares seven-bin HSV hue features using both current-to-first and current-to-previous similarities, with stronger decay to penalize long-range color drift.The metric balances global constancy with local transitions in the sequence.Source: D. Metrics details · p. 27
  • Sharpness Retention (SSharpness) tracks frame-to-initial-frame similarity of horizontal and vertical edge-gradient vectors, with a noise-triggered penalty to distinguish details from artifacts.The penalty mechanism triggers when BRISQUE noise exceeds a threshold for five consecutive frames, helping detect systematic visual collapse.Source: D. Metrics details · pp. 27–28, Figure 10 · p. 28
  • Motion Smoothness (SMotion) measures temporal continuity by reconstructing discarded odd frames with a video interpolation model and comparing them with the originals using LPIPS, SSIM, and MSE.The sampling-and-reconstruction approach is intended to capture instantaneous jitter or motion incoherence.Source: D. Metrics details · p. 28
  • Trajectory Accuracy (SAccuracy) scores alignment between ViPE-extracted, coordinate-aligned camera motion and control commands, while Trajectory Tolerance (STolerance) compares motion with system-built ground truth.Tolerance avoids third-party estimator uncertainty and serves as an upper-bound reference for controllability under ideal control conditions.Source: D. Metrics details · pp. 28, 30, Figure 11 · p. 29, Figure 12 · p. 29
  • Memory Symmetry (SMemory) scores pixel-wise consistency between symmetric frame pairs in cyclic or symmetric actions, weighting distant frames to detect memory decay in long sequences.It maps pairwise MSE to similarity and gives greater weight to frames near the sequence’s start and end.Source: D. Metrics details · p. 30
  • Trajectory Alignment (SAlignment) measures closed-loop camera-route symmetry by comparing direction vectors at matching points on the forward and reverse halves of round-trip trajectories.It assesses whether the model preserves spatial topology in go-and-return tasks.Source: D. Metrics details · p. 30, Figure 13 · p. 30

E.1. Human Preference Validation Details · E.2. Detailed comparison of world generation models

Human validation finds strong agreement between iWorld-Bench’s objective metrics and human preferences, significant and fine-grained model distinctions, and some rank discrepancies linked to metric weighting. The accompanying model-comparison subsection reports inference settings and generation-time measurement conditions.

E.1. Human Preference Validation Details

  • Validation paired Spearman correlation, Kruskal-Wallis testing, effect-size and 95% confidence-interval analysis, and Bonferroni-corrected Dunn comparisons to assess agreement, significance, magnitude, and fine-grained discrimination.The study’s questions concern metric-human rank agreement, perceptible differences across models, and discrimination among close-performing models.Source: §E.1 · p. 31
  • The study tested 14 models, including six close-performing systems, on 16 balanced tasks; each model received 192 ratings from 12 annotators on visual quality and camera-control precision.Tasks were sampled uniformly across four difficulty levels, and the study collected 2,688 ratings overall.Source: §E.1 · p. 31
  • 0.8053 (p = 0.0005) is the Spearman rank correlation between objective metrics and human preferences across 14 models; mean absolute rank difference is 1.93 positions.Table 11 presents the human and objective ranks and their differences.Source: Table 11 · p. 31, §E.1 · p. 32
  • H = 1496.8994 (p < 0.001; Ntotal = 2,688) and η2 = 0.5549 indicate significant, large human-perceived differences; 95% confidence intervals separate broad performance tiers.The top tier has mean scores ≥3.48, the mid tier 2.49–2.81, and the lower tier ≤2.26; limited overlap occurs among adjacent models within a tier.Source: Table 11 · p. 31, §E.1 · p. 32
  • 11 of 15 close-model pairs differ significantly (Bonferroni-corrected p < 0.001); four non-significant pairs have human-score differences ≤0.11, consistent with perceived ties.The four non-significant pairs are ASTRA–MotionCtrl, ASTRA–WAN 2.2, CameraCtrl–Matrix-game 2.0, and MotionCtrl–WAN 2.2.Source: §E.1 · p. 32, Table 12 · p. 32
  • AC3D’s |∆Rank| = 5 reflects strong Memory Symmetry and Motion Smoothness but low Image Quality, while NVIDIA Cosmos’s |∆Rank| = 3 combines strong visual quality with weaker trajectory accuracy and memory symmetry.The analysis suggests humans weight perceptual image quality more heavily than uniform-weight metric aggregation, motivating perceptually weighted aggregation.Source: §E.1 · p. 32, Table 13 · p. 32

E.2. Detailed comparison of world generation models

  • The model-comparison subsection reports inference configurations, variants, and per-instance generation times measured on NVIDIA A800 GPUs, as well as release dates and explicit camera-pose-input support.Table 14 is the detailed comparison of world-generation models evaluated in the benchmark.Source: §E.2 · p. 33, Table 14 · p. 33

Related papers

Loading 2605.03941v2…