PaperlayerView exact revision on arXiv

Source-linked AI summary

MBench: A Comprehensive Benchmark on Memory Capability for Video World Models

Shengjun Zhang, Zhang Zhang, Simin Huang, Zhenyu Tang, Hanyang Wang, Chensheng Dai, Min Chen, Yifan Li, Yuxin Li, Yingjie Chen, Hao Liu, Chen Li, Jing Lyu, Yueqi Duan

arXiv:2606.00793v2cs.CV

TL;DR

Existing video-world-model benchmarks largely overlook long-term memory, despite its importance for maintaining persistent world states. MBench evaluates this capability across entity, environment, and causal consistency, finding that current systems still struggle with stable entities, spatial layouts, controlled actions, and causal state changes. The results position memory as a first-class capability for future world models.

  • Problem

    Existing video-world-model benchmarks emphasize visual quality, motion coherence, and text-video alignment while overlooking long-term memory for maintaining consistent world states.

  • Method

    MBench decomposes video-world-model memory into entity, environment, and causal consistency and provides a granular evaluation beyond visual quality.

  • Results

    Current systems still struggle to preserve stable entities, recover consistent spatial layouts, follow controlled actions, and simulate causal state changes outside the visible field of view.

  • Takeaways & Limitations

    Memory should be treated as a first-class capability for future world models rather than as a by-product of visual fidelity or prompt alignment.

  • Takeaways & Limitations

    Causal memory remains unresolved, as current models often fail to continue hidden physical processes and produce plausible later states after out-of-view evolution.

Abstract

from arXiv · show

Recent advancements in video-based world models have demonstrated an unprecedented ability to synthesize high-fidelity visual sequences. However, a fundamental gap persists between visually plausible video generation and the functional requirements of a world model, particularly in maintaining a stable and reasonable internal state over extended temporal horizons. While existing benchmarks primarily emphasize visual quality, motion coherence, and text-video alignment, they largely overlook memory, the core capability of a world model to preserve consistency across long-term horizons and complex interactions. To address this gap, we present \textbf{MBench}, a comprehensive benchmark dedicated to quantifying and evaluating the memory capability of video world models. We systematically decompose the memory capability of video world models into three hierarchical and complementary core dimensions: entity consistency, environment consistency, and causal consistency, which are further refined into 12 quantifiable sub-dimensions for comprehensive characterization of long-term memory. Our benchmark is built upon rigorously curated real-captured long videos, and evaluated by rule-based quantitative matrices and VLM to enable objective and comprehensive consistency assessment. Extensive evaluations of mainstream state-of-the-art video world models reveal critical systemic limitations of existing methods in long-term state retention, providing a standardized benchmark and clear research direction to advance the field.

1 Introduction

MBench addresses the gap between visually plausible video generation and functional world modeling by benchmarking long-term memory for stable, consistent internal world states. It hierarchically evaluates entity, environment, and causal consistency using curated real-captured long videos and hybrid quantitative and VLM-based assessment.

  • Motivation: Existing benchmarks emphasize visual quality, motion coherence, and text-video alignment while overlooking long-term memory and structured consistency requirements.This leaves no standardized way to distinguish visually appealing frame generation from predictable real-world simulation.
  • Benchmark Design: MBench decomposes world-model memory into entity consistency, environment consistency, and causal consistency within a three-level hierarchical taxonomy.The framework is designed to systematically quantify and evaluate long-term memory capability.
  • Evaluation Dimensions: Entity consistency evaluates persistent identities and attributes of objects and humans, while environment consistency measures spatial layout, relative positioning, lighting, and stylistic coherence.These dimensions cover both individual participants and the scene stage in which interactions unfold.
  • Evaluation Dimensions: Causal consistency tests whether future dynamics preserve logical relationships with prior states and follow established physical and semantic rules.A destructive event should produce logically consistent subsequent states rather than reverting to an intact scene.
  • Evaluation Protocol: The benchmark combines rigorously curated real-captured long videos, contextual cues that expose memory failures, normalized heuristic metrics, and VLM-augmented VQA.Its taxonomy provides granular diagnosis from object-level details to scene-level layouts and from static-property persistence to dynamic temporal evolution.

2 Related Works

Related work progresses from short-clip video generation toward video world models, while recent methods increasingly treat long-horizon memory as an explicit design problem. Existing evaluations still do not jointly assess entity, environment, and causal consistency across long, text- and action-conditioned videos, motivating MBench.

  • Video Generation Models: Video generation evolved from GANs and VAE-based autoregressive transformers to diffusion models scaled through superresolution, latent diffusion, and diffusion transformers.Early systems produced short, low-resolution clips, whereas Sora demonstrated minute-level photorealistic synthesis through large-scale DiT training.
  • World Models: World models learn environment dynamics and predict future states, extending from reinforcement-learning settings through recurrent, transformer, and diffusion-based dynamics to open-domain video pretraining.Applications include interactive game-style worlds and driving simulators, among other application-driven lines.
  • Memory for Video World Models: Long-horizon consistency is the defining challenge, prompting memory architectures that store world-state information in context, external banks, or model parameters.Earlier approaches used causal masking, hierarchical diffusion, and temporal co-denoising but treated history uniformly and left consistency to emerge implicitly.
  • Memory for Video World Models: In-context memory compresses or streams history, external memory retrieves stored information on demand, and parametric memory adapts model weights or training schemes to handle imperfect history.Compression allocates more tokens to recent frames, streaming uses sliding-window autoregression, and external retrieval can use camera pose, semantic relevance, or hierarchical descriptors.
  • Benchmarking Memory: Existing benchmarks measure realism, text–video alignment, perceptual quality, physical fidelity, instruction following, or memory under specific conditions, but do not jointly cover the three consistency axes and required conditioning settings.MBench addresses this gap with a three-axis memory taxonomy and Trigger-Condition...

3 Benchmark Construction

MBench constructs a long-term memory benchmark from diverse real-world videos spanning varied environments, interactions, camera motions, and physical transitions. Its evaluation protocols use structured descriptions and memory-triggering viewpoint changes tailored to video continuation and action-conditioned world models.

  • Data Collection: MBench aggregates five real-world video datasets covering indoor and outdoor environments, human-object interactions, dynamic camera motions, and physical state transitions.The datasets are DL3DV, Tanks and Temples, OpenHumanVID, SpatialVID, and Physics-aware-video.
  • Data Collection: Video durations range from 5 seconds to 15 minutes, providing temporal context for evaluating long-term memory retention.The duration range is reported across the aggregated datasets.
  • Challenge Curation: A VLM selects challenging videos by assessing entity consistency, environment consistency, and related memory-consistency challenges.Entity challenges include occlusion, attribute changes, and complex motion trajectories; environment challenges include camera motion, scene-layout changes, and dynamic backgrounds.
  • Structured Descriptions: Each curated clip receives a structured description of scene activity, entity attributes and positions, and the chronological sequence of key events.These descriptions provide the temporal and spatial information used by subsequent evaluation protocols.
  • Video Continuation Protocol: For video continuation models, descriptions are divided into five consecutive temporal paragraphs and paired with camera instructions involving translation, rotation, zooming, and dynamic occlusion.The instructions are designed to test state consistency under changing viewing conditions.
  • Action-Conditioned Protocol: For action-conditioned world models, the Exit-and-Reenter paradigm tests whether models retain complete entity states while targets temporarily leave the camera’s field of view.The protocol uses an initial frame I0, a predefined target entity E, and a standardized action sequence A.

4 Benchmark Evaluation

MBench evaluates video world-model memory through a multi-dimensional automated pipeline covering entity, environment, and causal consistency. Its trigger-conditioned scoring separates instruction responsiveness from consistency under successfully activated memory challenges, while specialized metrics assess persistent entities, scene conditions, and physically plausible evolution.

  • Evaluation Framework: The evaluation pipeline decomposes memory capability into entity consistency, environment consistency, and causal consistency, using specialized quantitative metrics for fine-grained assessment.This hierarchical design is intended to provide a comprehensive and interpretable evaluation of model memory.
  • Trigger-Conditioned Scoring: Trigger-Conditioned Scoring verifies whether each sample activates its memory challenge before computing consistency, excluding untriggered samples from memory evaluation.A VLM records binary trigger labels for events such as object occlusion or camera-induced entity exit.
  • Trigger-Conditioned Scoring: The final M-Score uses the harmonic mean of Srel and trigger coverage, penalizing conservative models that avoid memory challenges while rewarding consistency during frequent state transitions.Ctrig measures triggered-sample coverage, whereas Srel averages consistency only over successfully triggered samples.
  • Entity Consistency: Entity consistency measures persistent identity, attributes, and trajectories, including geometry after departure-return occlusions, texture stability, and human identity across temporal gaps.Geometry uses viewpoint-aligned frame pairs and object masks, texture uses DINOv2 features, and identity uses ArcFace-based tracking over uniformly sampled frames.
  • Environment and Causal Consistency: Environment and causal evaluation test whether models preserve lighting when revisiting scenes and autonomously produce physically correct state evolution after instructed events.Self-Evolution requires both successful event actuation and physically sound progression, while Lighting Consistency evaluates temporal stability of brightness, color temperature, and color distribution.

5 Experiments

Experiments evaluate text-conditioned continuation and action-conditioned world models on MBench, revealing persistent memory bottlenecks despite visually plausible generation. Results show complementary strengths across models, with substantial weaknesses in spatial, style, and causal consistency.

  • Evaluation setup: MBench evaluates text-conditioned continuation models and action-conditioned world models under iterative extension and continuation protocols.Text-conditioned baselines include Memflow, Self Forcing, Skyreels V2, Longlive, Longcat-Video, Cosmos-Predict 2.5, Causal Forcing, and Helios.
  • Overall findings: Visually plausible videos often fail to preserve persistent world state after target entities or camera views leave the visible region.The failure is especially pronounced in spatial and causal metrics requiring recovery of previously unseen state.
  • Text-conditioned models: Self Forcing and LongLive are the strongest text-conditioned baselines across several memory dimensions, while Helios leads object geometry and texture.Self Forcing leads human identity and most environment metrics; LongLive leads prompt interaction, and Causal Forcing leads human appearance.
  • Text-conditioned models: Style consistency remains weak for all continuation models, even when epipolar and lighting consistency are comparatively stronger.The results indicate that stable visual tone does not ensure retention of the underlying 3D layout.
  • Action-conditioned models: HY-WorldPlay achieves the strongest overall entity and rendering consistency among action-conditioned models and leads epipolar and reprojection scores.Infinite-World and Matrix-Game 3.0 follow HY-WorldPlay on spatial metrics, whereas Matrix-Game 2.0 and Lingbot-World show larger geometric errors.
  • Action-conditioned models: HY-WorldPlay and Infinite-World excel in rendering and spatial consistency but perform poorly on self-evolution, indicating overly static scene generation.Their state evolution scores are markedly lower than those of Yume-1.5 and Lingbot-World.
  • Human correlation: Geometry consistency reaches ρ = 0.97 for text-conditioned continuation, while epipolar geometry reaches ρ = 0.94 for action-conditioned models.Lighting and style correlate at ρ ≈0.71 for action-conditioned rollouts, compared with ρ = 0.16 for action-conditioned epipolar geometry in continuation models.

6 Discussion

Current long-video generation systems remain unreliable as video world models because persistent world states drift over extended horizons. The main unresolved weaknesses are limited spatial reasoning, unstable action-conditioned control, and failure to model meaningful causal state transitions.

  • Overall limitations: Persistent world-state coherence often breaks after occlusion, camera motion, or long-horizon continuation, despite locally convincing entities, backgrounds, and causal states.The systems can synthesize visually plausible and temporally extended clips, but their apparent coherence does not reliably persist.
  • Spatial understanding: Spatial reasoning remains limited because models may misfollow camera-motion instructions, while vision models struggle to infer camera motion from sparse frames.Generated motion can move in the wrong direction, stop too early, or diverge from the requested trajectory.
  • Action-conditioned control: Action-conditioned models provide unstable control, sometimes delaying responses, ignoring control signals, or collapsing to frequent training-time motion patterns.A rollout may appear reasonable as generic video while failing to function as an interactive world simulation.
  • Causal memory: Causal memory is the central unresolved challenge because many action-conditioned models preserve appearances while failing to simulate meaningful dynamic state transitions.Generated scenes may remain mostly static and still receive acceptable appearance or rendering-consistency scores.

7 Conclusion · Appendix

MBench is introduced as a benchmark for evaluating video world-model memory across entity, environment, and causal consistency, with conclusion findings that current systems struggle with stable long-horizon state. The appendix provides further details on benchmark construction and evaluation protocols for both evaluation settings.

  • 7 Conclusion: MBench evaluates video world-model memory across entity, environment, and causal consistency.The benchmark is presented as a dedicated evaluation framework for these three consistency dimensions.
  • 7 Conclusion: MBench tests persistent world-state retention under long-horizon continuation, camera motion, occlusion, action conditioning, and hidden causal evolution.These conditions distinguish its evaluation from standard video-generation assessment.
  • 7 Conclusion: Current systems struggle to preserve stable entities, recover consistent spatial layouts, and follow contr…The supplied conclusion passage truncates the final finding after “follow contr”.
  • 7 Conclusion: The evaluations cover both text-conditioned long-video generators and action-conditioned world models.The conclusion explicitly reports evaluation across both model settings.
  • 7 Conclusion: The work acknowledges support from Chinese national, Beijing municipal, and CAST Young Elite Scientist Sponsorship Program grants.The listed grants include 62576185, L252011, and YESS20240544.
  • Appendix: The appendix details MBench construction and evaluation protocols for both evaluation settings.It introduces dataset composition, evaluation and trigger mechanisms, computational-strategy differences between MBench-T and MBench-A, and human annotation protocol.

A Dataset and Prompt Details

MBench is constructed from five real-world video datasets spanning diverse scenarios and organized into evaluation subsets for environment, entity, and causal consistency. Its prompt and data design maps human and object subsets to entity dimensions, while environment and causal subsets assess their corresponding consistency dimensions.

  • A.1 Data Composition: MBench is built from five real-world video datasets covering diverse scenarios.
  • A.1 Data Composition: The Human subset evaluates the Entity-Human dimension.
  • A.1 Data Composition: The Object subset evaluates the Entity-Object dimension.
  • A.1 Data Composition: Table 4 summarizes the number of evaluation samples for each subset across two benchmark settings.
  • A.1 Data Composition: DL3DV, Tanks and Temples, and SpatialVID provide data for the environment and object subsets.
  • A.1 Data Composition: OpenHumanVID constructs the human subset, while Physics-aware-video contributes to the causal subset.

B Evaluation and Trigger Details

MBench uses shared core metric formulas across text-conditioned video continuation and action-conditioned world models. Their preprocessing differs in frame or segment selection, memory-trigger determination, and target-entity detection.

  • The 12 sub-dimensions share a core metric formula across text-conditioned video continuation and action-conditioned world models.
  • Preprocessing differs in how frame pairs or segments are selected for evaluation.
  • Preprocessing also differs in how the memory trigger is determined and how the target entity is detected.

B.1 Pair Selection Strategies … C.2 Annotation Statistics

MBench combines geometry-aware pair selection, challenge-trigger verification, and entity-tracking pipelines to evaluate long-term memory in text- and action-conditioned videos. Human annotations provide trigger validation and majority-voted pairwise preferences across 12 memory dimensions.

  • B.1 Pair Selection Strategies: MBench-T selects frame pairs from early and late caption segments, ranking candidates by pose-return similarity and then temporal span.Pose-return similarity uses DA3 extrinsics and combines translation and rotation differences.
  • B.1 Pair Selection Strategies: MBench-A detects the camera turnaround point, matches outbound frames to inbound frames by combined translation and rotation distance, and retains up to 20 longest-span pairs.Fixed symmetric pairs are used when no clear turnaround is detected.
  • B.2 Trigger Mechanism: For MBench-T, a VLM trigger judge uses eight uniformly sampled frames and caption segments to verify whether each generated video activates its intended memory challenge.The judge covers human, object, environment, and causal challenge types, returning a binary decision with confidence.
  • B.2 Trigger Mechanism: A sample is evaluable only when the trigger judge returns trigger, while low trigger coverage is penalized through the M-Score harmonic mean.MBench-A omits semantic triggering because explicit spatial trajectories deterministically enforce occlusion and reappearance.
  • B.3 Entity Detection Details: MBench-T detects target objects with Grounding DINO and initializes SAM 2 masks from the resulting boxes before propagating them through the video.Each item supplies a semantic profile containing a descriptive name and core visual attributes.
  • B.3 Entity Detection Details: MBench-A uses ground-truth spatial anchors to initialize SAM 2 masks, propagates them across sampled frames, and applies InsightFace to detect human faces.Anchors specify center coordinates, width, and height in the first frame.
  • C.1 Annotation Protocol: Human annotation uses separate trigger and metric-preference interfaces for validating memory-challenge activation and comparing outputs on specific consistency properties.The preference interface presents two outputs side by side and allows a better, worse, or tie judgment for each evaluated sub-dimension.
  • C.2 Annotation Statistics: Pairwise comparisons cover 12 fine-grained memory dimensions, with three independent annotators per video pair and majority consensus as the ground-truth preference.Complete ties are identified when all three annotators provide divergent responses.

C.3 Validation of VLM-based Triggers · D More Visualizations and Case Studies · D.1 Trigger Mechanism Visualization

The paper validates its VLM-based trigger judge against human annotations and presents visual case studies of trigger behavior. The validation emphasizes high recall, while the visualizations illustrate successful and failed memory-challenge triggering.

  • C.3 Validation of VLM-based Triggers: 400 human annotations across causal, object, environment, and human subsets evaluate whether generated videos visibly enter the intended memory challenge.The annotations provide majority-vote labels for assessing VLM trigger decisions.
  • C.3 Validation of VLM-based Triggers: The validation compares VLM trigger decisions with human majority votes using accuracy, precision, recall, and F1-score.These four metrics quantify agreement between automated trigger judgments and human labels.
  • C.3 Validation of VLM-based Triggers: ≥0.73 recall across all subsets indicates that the VLM rarely misses genuine memory challenges.The benchmark treats false negatives as especially undesirable because they exclude informative samples from evaluation.
  • C.3 Validation of VLM-based Triggers: 0.90 overall recall supports using the VLM as an effective gatekeeper for downstream benchmark metric evaluation.False positives admit samples into downstream evaluation, whereas false negatives exclude genuinely informative data.
  • D More Visualizations and Case Studies: The visualizations illustrate trigger effectiveness and the discriminative power of MBench metrics across entity, environment, and causal memory dimensions.This section provides qualitative evidence alongside the quantitative benchmark evaluation.
  • D.1 Trigger Mechanism Visualization: Figure 5 uses two contrastive examples generated from the same five-stage caption sequence about a woman leaving and returning to an office.The shared prompt sequence isolates differences in trigger behavior between the examples.
  • D.1 Trigger Mechanism Visualization: Example 5a successfully triggers the memory challenge by producing the required disappearance–reappearance event despite relatively weak appearance preservation.The example faithfully follows the prompt sequence even though visual identity preservation is comparatively weak.
  • D.1 Trigger Mechanism Visualization: Example 5b is presented as failing to trigger the memory challenge under the same five-stage prompt sequence.Its contrast with 5a demonstrates the trigger mechanism’s sensitivity to whether the intended challenge is realized.

D.2 Metric Scores Visualization

Qualitative visualizations show that MBench’s metrics discriminate successful from failed videos under identical prompts or action sequences across entity, environment, and causal consistency. The metrics identify detail preservation, spatial and rendering stability, physically plausible evolution, and adherence to long-term action or text instructions.

  • Overall visualization: Under identical conditions, the metrics distinguish high-scoring successful videos from low-scoring failed videos across all three core memory dimensions.Each good–bad pair uses the same input prompt or action sequence.
  • Entity Consistency: Entity consistency scores separate preserved facial and geometric details from identity drift, garment changes, facial distortion, and object deformation during departure–return trajectories.Human consistency uses appearance and identity, while object consistency uses geometry and texture.
  • Environment Consistency: Environment consistency scores capture preserved 3D layouts and global illumination versus geometric collapse, misaligned revisits, and stylistic hallucinations.Spatial consistency measures reprojection and epipolar alignment, while rendering consistency measures style and lighting.
  • Causal Consistency: Causal consistency heavily penalizes physically implausible transitions, including morphing a shredder into foliage instead of destroying a deodorant can.The low-scoring example receives 0.00, while the physically correct high-scoring example receives 100.00.
  • Causal Consistency: The metrics penalize noisy action control and semantic scene drift during long-term sequential execution.Action adherence scores 75.53 versus 57.78 for tightly followed versus noisy trajectories, while text adherence scores 18.73 versus 29.49 for failed versus successful five-segment instruction following.
Loading 2606.00793v2…