Source-linked AI summary
MIND: Benchmarking Memory Consistency and Action Control in World Models
Yixuan Ye, Xuanyu Lu, Yuxin Jiang, Yuchao Gu, Rui Zhao, Qiwei Liang, Jiachun Pan, Fengda Zhang, Weijia Wu, Alex Jinpeng Wang
TL;DR
World-model evaluation lacks a unified benchmark for memory consistency and action control in dynamic, open-domain environments. MIND introduces a closed-loop revisited benchmark with multi-view, action-aligned videos and an evaluation framework covering memory, control, and action-space generalization. Experiments with MIND-World reveal persistent difficulties in long-term memory consistency, open-domain generalization, and cross-action-space generalization.
Problem
Current evaluations emphasize visual quality or physical realism, leaving comprehensive assessment of long-term memory consistency and action controllability in open-domain environments unresolved.
Method
MIND provides 250 frame-level action-aligned videos across eight scene categories and evaluates memory consistency, action control, and generalization across varied action spaces and viewpoints.
Results
Experiments reveal that current world models struggle with long-term memory consistency, open-domain generalization, and generalization across changed action spaces.
Takeaways & Limitations
MIND establishes a unified foundation for evaluating and advancing interactive, temporally consistent open-domain world models.
Abstract
from arXiv · showhide
World models aim to understand, remember, and predict dynamic visual environments, yet a unified benchmark for evaluating their fundamental abilities remains lacking. To address this gap, we introduce MIND, the first open-domain closed-loop revisited benchmark for evaluating Memory consIstency and action coNtrol in worlD models. MIND contains 250 high-quality videos at 1080p and 24 FPS, including 100 (first-person) + 100 (third-person) video clips under a shared action space and 25 + 25 clips across varied action spaces covering eight diverse scenes. We design an efficient evaluation framework to measure two core abilities: memory consistency and action control, capturing temporal stability and contextual coherence across viewpoints. Furthermore, we design various action spaces, including different character movement speeds and camera rotation angles, to evaluate the action generalization capability across different action spaces under shared scenes. To facilitate future performance benchmarking on MIND, we introduce MIND-World, a novel interactive Video-to-World baseline. Extensive experiments demonstrate the completeness of MIND and reveal key challenges in current world models, including the difficulty of maintaining long-term memory consistency and generalizing across action spaces. Code: https://github.com/CSU-JPG/MIND.
1. Introduction
MIND addresses the lack of a comprehensive open-domain benchmark for world-model memory consistency and action control. It combines multi-view, action-aligned videos with evaluations of temporal stability, contextual coherence, and cross-action-space generalization.
- Existing evaluations emphasize visual quality or physical realism while overlooking long-term memory consistency and action controllability in open-domain environments.
- MIND is an open-domain closed-loop revisited benchmark evaluating memory consistency and action control from first-person and third-person perspectives.Memory consistency concerns preserving spatial layouts, object identities, and scene attributes over long contexts; action control concerns executing inputs and generalizing to new motion ranges or action spaces.
- 250 high-quality 1080p / 24 FPS videos span eight scene categories and include frame-level action, character-position, camera-position, and image-label annotations.The videos were collected from multiple volunteers to capture diverse human behaviors.
- MIND evaluates temporal stability and contextual coherence across viewpoints and tests action-space generalization using different character movement speeds and camera rotation angles.
- MIND-World provides a novel video-to-world baseline, while experiments expose limited long-term memory consistency and limited generalization across action spaces in current world models.
2. Related Work
World-model research has progressed from realistic video generation toward interactive, temporally consistent environments. Existing benchmarks evaluate video quality, physical plausibility, or realism, but underrepresent long-context memory consistency and action-space generalization.
- 2.1. Video Generation: Recent video-generation models improve visual realism, temporal consistency, controllability, and physically plausible long-sequence generation.
- 2.1. Video Generation: VBench and VBench-2.0 evaluate video-generation dimensions including human fidelity, physical plausibility, and commonsense consistency.
- 2.2. World Model: World models extend visual generation toward embodied simulation and interaction in temporally consistent virtual environments.
- 2.2. World Model: Diffusion-Forcing and Self-Forcing support interactive streaming generation through per-token noise conditioning and autoregressive inference with KV caching.
- 2.3. World Model Benchmarks: WorldScore, WorldModelBench, and WorldSimBench primarily assess camera-trajectory video quality, physical-law adherence, or visual realism, underrepresenting long-context memory and action-space generalization.
3. MIND Benchmark
MIND is an open-domain benchmark built from diverse Unreal Engine 5 environments to evaluate memory consistency, action control, and action-space generalization in world models. Its framework combines revisited trajectories, long-context reconstruction, symmetric-path consistency, action accuracy, and visual-quality evaluation across viewpoints and scenes.
- Video Source and Environment Settings: MIND spans 8 categories covering over 40 open-domain environments, including natural, urban, indoor, vehicle, sci-fi, stylized, fantasy, and abstract scenes.The corpus is rendered in Unreal Engine 5 and uses scripted and free-form actions performed by multiple volunteers.
- Basic Actions Modeling: The benchmark models agent translation with W, A, S, D and camera pitch or yaw with ↑, ↓, ←, →.Translational actions use direction vectors and step size ∆p, while camera actions use angular increments ∆r.
- Basic Actions Modeling: MIND varies ∆p and ∆r to create action spaces ranging from precise small steps to broader movements across first-person and third-person clips.The benchmark includes 25 first-person and 25 third-person clips for systematic cross-action-space evaluation.
- Temporal and Memory Consistency: The memory-revisit setup gives models an observed sequence and subsequent actions, then evaluates whether predicted frames preserve previously seen objects, layouts, textures, and temporal coherence.Ground-truth frames correspond to revisited scenes, enabling comparison against memorized content.
- Temporal and Memory Consistency: Long-context memory uses mean squared error between predicted and ground-truth frames, with lower Llcm indicating stronger retention and reconstruction fidelity.The setup evaluates reconstruction after a full memory sequence and an action sequence.
- Evaluation: Generated scene consistency compares forward and reverse predictions along 10 symmetric 24-frame motion paths, where lower Lgsc indicates stronger geometric stability.The action sequence moves along a path and retraces it in reverse, so corresponding generated frames should match.
- Evaluation: Action accuracy aligns predefined actions across models, recovers camera trajectories, applies Sim(3) Umeyama alignment, and measures translational and rotational relative pose errors.Action-space generalization separately compares generated and ground-truth frames under diverse action settings, while visual quality uses aesthetic and related measures.
4. Experiment
MIND-World provides an interactive Video-to-World baseline evaluated on first- and third-person MIND settings, exposing memory, action-control, and generalization challenges. Experiments show benefits from context memory for long-context performance, but action-space changes and long-horizon interaction remain difficult.
- MIND-World: MIND-World injects actions directly into timestep embeddings and uses cached context for continuous, low-latency autoregressive frame generation.The framework supports context-memory and cold-start inference settings and is designed for unlimited-length inference from action sequences.
- Per-Dimension Evaluation: More than 4%: context-memory models outperform models without context memory on the long-context memory metric.Generated-scene consistency results further confirm the benefits of memory.
- Per-Dimension Evaluation: Action control deteriorates even with context memory and the same action space used during fine-tuning, highlighting limitations in current action injection.The authors identify more effective action-injection strategies as an open research problem.
- Insights and Discussions: MIND-trained models generalize better to open-domain inference than models trained on easily collected Minecraft data, but acquiring suitable data remains challenging.The authors identify effective use of readily available large-scale data for open-domain generalization as an open problem.
- Insights and Discussions: Changing action spaces significantly harms memory-enabled inference because inconsistent action spaces disrupt model reasoning.Within the original action space, memory-enabled models outperform memory-less models, but performance drops when the action space changes.
- Insights and Discussions: Long-context memory, generated-scene consistency, precise action control, and third-person character-background interactions remain major challenges.Examples include failures to preserve revisited scenes, execute return movements, or maintain character-background relationships.
5. Conclusion
The paper introduces MIND as an open-domain closed-loop revisited benchmark for memory consistency and action control across first- and third-person perspectives. Experiments with MIND-World show that generalizing across action spaces and maintaining long-horizon coherence remain challenging.
- 5. Conclusion: MIND evaluates long-term scene memory, temporal coherence, and action-space generalization using diverse action spaces in Unreal Engine 5.It supports systematic evaluation from both first-person and third-person perspectives.
- 5. Conclusion: Experiments reveal persistent challenges in generalizing across action spaces and maintaining long-horizon coherence.These findings define central remaining difficulties for interactive, temporally consistent open-domain world models.
- 5. Conclusion: MIND establishes a unified foundation for advancing interactive, temporally consistent open-domain world models.The conclusion positions the benchmark as a basis for future progress in this setting.