Source-linked AI summary
MemoBench: Benchmarking World Modeling in Dynamically Changing Environments
Haoyu Chen, Kaichen Zhou, Hang Hua, Kaile Zhang, Jingwen Qian, Wufei Ma, Haonan Chen, Chunjiang Liu, Yizhou Zhao, Xiaoyuan Wang, Weiyue Li, Alan Yuille, Paul Pu Liang, Yilun Du
TL;DR
Existing benchmarks under-examine visual memory when objects disappear amid changing environments. MemoBench evaluates this setting across 360 clips and finds that no current model reliably maintains object memory across occlusion, with no model exceeding an Object Reappearance Score of 0.6.
Problem
Visual memory under partial observability remains under-explored, particularly when objects leave view while environments continue changing.
Method
MemoBench evaluates disappear-and-reappear memory consistency across 360 clips using automated metrics and VQA across four diagnostic dimensions.
Results
No model exceeds an Object Reappearance Score of 0.6, indicating that current models struggle to reliably recover objects after occlusion.
Takeaways & Limitations
Object permanence does not emerge from trajectory control alone, indicating that memory must be explicitly addressed in model design.
Takeaways & Limitations
Standard consistency metrics can be inflated by camera inactivity and may not distinguish genuine reappearance from static viewpoints.
Abstract
from arXiv · showhide
Video generation models aspire to simulate dynamic environments, and several benchmarks now evaluate memory consistency across frames. However, most assess consistency only while the target remains in view, and the few that force objects out of view evaluate static scenes where nothing changes during occlusion. To bridge this gap, we introduce MemoBench, a diagnostic benchmark built around the disappear-and-reappear paradigm in dynamically changing environments: a target object undergoes a physical process, disappears from view, and must be correctly recovered in its updated state upon reappearance. We curate 360 ground-truth clips spanning synthetic and real-world scenes, and design an evaluation suite combining automated metrics with VQA-based assessment across four diagnostic pillars. Evaluation of eight state-of-the-art models reveals key insights and open challenges regarding memory consistency under the disappear-and-reappear paradigm.
1 Introduction
MemoBench addresses the under-explored challenge of maintaining and updating object states when targets disappear from view amid ongoing physical and camera dynamics. It introduces a benchmark and evaluation suite spanning diverse synthetic and real-world state changes, revealing that current models do not reliably preserve object memory across occlusion.
- Motivation: Dynamic environments require models to reason about how the world changes beyond their field of view, supporting applications such as autonomous driving, robotic manipulation, and embodied tasks.Examples include melting ice, flickering flames, walking pedestrians, and flowing traffic.
- Problem: Visual memory under partial observability remains under-explored, despite object permanence being a foundational cognitive capability.The benchmark frames model memory as analogous to understanding that objects continue to exist when out of sight.
- Problem: Existing benchmarks mostly evaluate continuously visible content and do not directly test state maintenance under simultaneous camera and scene dynamics.This leaves unclear whether models genuinely remember temporarily hidden objects or merely regenerate scene content.
- Method: MemoBench uses a disappear-and-reappear sequence in which a visible target undergoes a physical process, remains hidden while the process continues, and reappears in an updated state.The camera first views the target, pans away during the ongoing process, and then returns to evaluate state recovery.
- Benchmark: 360 high-quality ground-truth videos at 1920×1080 resolution span diverse scenes and physical-state changes, forming the benchmark’s core dataset.The collection includes 196 synthetic and 164 real-world clips.
- Evaluation and findings: The evaluation combines automated metrics and LLM-judged VQA across four diagnostic dimensions, and benchmarking ten state-of-the-art models finds that no current model reliably maintains object memory across occlusion.Automated measures include video quality, Object Reappearance Score, pixel-level fidelity, and camera controllability.
2 Related Work
Related work has advanced video generation toward world simulation and developed benchmarks for scene traversal, camera control, and scene consistency. MemoBench addresses their stated gap by evaluating object permanence when both viewpoints and scene content change through disappear-and-reappear sequences.
- Video Generation as World Simulation: Video generation has progressed from short-clip synthesis toward world simulation involving physics, causal dynamics, and persistent environmental state.Notable models include CogVideoX, Open-SoRA, LTX-Video, and LingBot-World, which explicitly targets long-term memory.
- MemoBench: MemoBench is the only benchmark explicitly evaluating memory consistency through the disappear-and-reappear paradigm.Its paradigm requires models to maintain memory while a target object undergoes changing conditions outside the view.
- Related Evaluation Settings: World-in-World evaluates closed-loop embodied world models by task-level success rather than fine-grained visual consistency of individual objects.Other related work includes models that model trajectories without testing object permanence.
- Benchmarking World Models: Existing benchmarks advance scene traversal, camera control, and scene consistency but do not jointly test dynamic viewpoints and evolving scene content.The missing capability is maintaining a target object’s evolving state after disappearance and recovering it correctly upon reappearance.
3 MemoBench
MemoBench is a 360-clip benchmark spanning synthetic spatial diversity and real-world material diversity for evaluating disappear-and-reappear video generation. Its evaluation combines phase-aware automated metrics with multi-stage VQA assessment.
- Dataset: MemoBench contains 360 ground-truth clips: 196 synthetic clips across 14 scene subdomains and five environment categories, plus 164 real-world clips covering 30 processes across seven categories.The synthetic subset emphasizes spatial diversity and egomotion, while the real-world subset emphasizes material properties such as viscosity, elasticity, and thermal conductivity.
- Evaluation protocol: Each clip is divided into Visible, Disappeared, and Reappear phases using annotated disappearance and reappearance keyframes.The Disappeared phase is excluded by design from motion and geometry metrics unless otherwise specified.
- VQA-based evaluation: A complementary multi-stage VQA metric uses an LLM evaluator, Gemini-3.1-Pro, to assess multimodal generation quality.The VQA evaluation is organized around diagnostic dimensions and uses Yes/No questions.
- Automated metrics: Automated evaluation measures visual quality, motion smoothness, object identity consistency, and Geo3D consistency across generated videos.These metrics use AestheticScore, CLIP-IQA+, RAFT-Large optical flow, DINOv2 patch-token similarity, and Depth Anything V2 depth-map similarity.
- Automated metrics: The benchmark additionally evaluates reappearance reliability with Object Reappearance Success, based on valid SAM-3 detections and their confidence scores during the Reappear phase.Coverage filtering rejects spurious large-area masks and noise; high ORS indicates reliable regeneration of a recognizable target object.
- Automated metrics: Pixel-level fidelity is reported with PSNR (↑), SSIM (↑), and LPIPS (↓) separately for Visible, Disappeared, Reappear, and full-video phases.The suite also reports ImageReward Score for prompt fidelity after mapping raw scores to [0, 1] and then [0, 100].
4 Evaluation Results
Evaluation shows that conditioning interface and camera activity strongly affect apparent consistency, while geometric fidelity, perceptual quality, and object memory remain misaligned. Across automated and VQA assessments, no model reliably combines precise trajectories, faithful reappearance, and semantic correctness.
- Evaluation setup: Ten models are evaluated across control-conditioned and text-image-to-video categories on MemoBench.The evaluated set includes seven C+TI2V models and three open-source TI2V models.
- Automated evaluation: Direct camera-pose conditioning improves trajectory precision, whereas action-conditioned dynamics do not translate prescribed trajectories into equally precise camera motion.Stable Virtual Camera follows target poses by construction, while Matrix-Game 2.0 receives action inputs rather than explicit camera poses.
- Automated evaluation: LTX-Video reaches an ORS of 0.330 despite low camera activity, showing that nearly static viewpoints can inflate consistency and memory metrics.Consecutive near-identical frames trivially maximize flow-based smoothness, depth consistency, identity similarity, and ORS.
- Automated evaluation: No model exceeds an ORS of 0.6, and no single model combines strong Camera Controllability, high ORS, and competitive Visual Quality.LTX-Video’s apparent object-memory success reflects a static viewpoint, while even strong ORS performers can fail to recover the target faithfully.
- Semantic evaluation: VQA exposes semantic failures missed by automated metrics, with Matrix-Game 2.0 scoring lowest on Object & Background and Physics Adherence despite high Visual Quality.Stable Virtual Camera similarly combines strong pixel-level fidelity with below-average VQA scores because artifacts such as warping seams and texture flickering affect semantic correctness.
5 Conclusion
MemoBench evaluates memory consistency in world generation through disappear-and-reappear scenarios, combining automated metrics and VQA across 360 clips. Results show that current models struggle to preserve object representations after disappearance, with no model exceeding an Object Reappearance Score of 0.6.
- Conclusion: MemoBench evaluates memory consistency in world generation through the disappear-and-reappear paradigm.The benchmark combines automated metrics with a VQA pipeline across 360 clips.
- Conclusion: 360 clips are assessed using automated metrics and a VQA pipeline.The evaluation spans the benchmark’s world-generation memory-consistency setting.
- Conclusion: 0.6 is the highest Object Reappearance Score achieved by any model.The result indicates that current models struggle to maintain persistent object representations after objects leave the field of view.
A Supplementary Materials · A.7.1 ORS Robustness Analysis. · A.1 Model Configurations
The evaluation uses ten open-source models across control-conditioned and standard text-image-to-video generation, with configurations covering resolution, frame rate, video length, and camera-conditioning support. Camera-conditioned models differ in whether they accept camera trajectories, pose conditioning, or action inputs.
- A.1 Model Configurations: Ten open-source models span control-conditioned text-image-to-video generation (C+TI2V) and standard text-image-to-video generation (TI2V).C+TI2V and TI2V denote control-conditioned and standard text-image-to-video generation, respectively.
- A.1 Model Configurations: Five C+TI2V models—LingBot-World, Wan2.2, FantasyWorld, HunyuanWorldPlay, and HunyuanGameCraft—accept camera trajectories as input.Camera trajectories provide direct control over viewpoint changes.
- A.1 Model Configurations: Stable Virtual Camera is pose-conditioned but targets static-scene novel view synthesis without text conditioning or temporal dynamics.Its training objective differs from dynamic text-conditioned video generation.
- A.1 Model Configurations: Matrix-Game 2.0 receives control through action inputs rather than explicit camera poses.Its control interface therefore differs from models that accept camera trajectories.
- A.1 Model Configurations: Table S1 summarizes each model’s output resolution, frame rate, generated video length, and camera-conditioning support.The table describes the model configurations used in the evaluation.
- A.1 Model Configurations: All evaluated models are open-source baselines with configurations summarized in Table S1.The table organizes their resolution, frame rate, video length, and camera-conditioning support.
A.2 Implementation Details
The experiments reproduce each baseline using official codebases and public checkpoints on a four-GPU server, with model-specific inference configurations summarized for three video generation models.
- Hardware and baseline reproduction: Experiments run on four NVIDIA RTX Pro 6000 GPUs, reproducing each baseline with its official codebase and publicly available checkpoints.The authors state that the following configurations summarize model inference settings.
- Model-specific inference configurations: LingBot uses the lingbot-world-base-cam checkpoint to generate 81 frames at 464×832 resolution with UniPC, 70 sampling steps, guidance scale 5.0, and 16 fps.These settings define the reported LingBot inference configuration.
- Model-specific inference configurations: FantasyWorld uses Wan2.1-I2V-14B-480P to generate 81 frames at 336 × 592 resolution with 50 flow-matching steps and guidance scale 5.0.These settings define the reported FantasyWorld inference configuration.
A.3 Dataset Statistics
MemoBench comprises 360 clips split between diverse synthetic and real-world subsets, covering varied scene domains, physical-state changes, camera trajectories, and temporal gaps. Synthetic U-turn sequences impose the greatest viewpoint and temporal challenge, with 178° mean rotation and 113-frame gaps.
- 196 synthetic clips span 14 scene subdomains across five environment categories, with sequences typically lasting 260–300 frames at 1920×1080 and 60 FPS.
- 164 real-world clips cover 30 physical-state-change processes grouped into seven major categories, with 103–349-frame sequences captured at 1920×1080.
- Synthetic trajectories generally feature larger viewpoint changes and longer temporal gaps than real-world trajectories, which primarily use controlled horizontal pans and vertical tilts.
- 178° mean camera rotation and 113-frame mean departure-to-reappearance gaps make U-turn sequences the most challenging setting for memory across substantial viewpoint changes.
A.4 More Dataset Examples
Additional examples illustrate MemoBench’s synthetic and real-world subsets, spanning V-D-R sequences and seven real-world state-change categories with transformations occurring off-screen during camera departure.
- Synthetic Data: Synthetic examples sample five uniformly spaced frames across the visible, departure, and return phases of each clip.The V-phase shows the target, the D-phase captures camera departure, and the R-phase captures camera return.
- Real-World Data: Real-world examples cover seven state-change categories, with the camera panning away while physical transformations occur off-screen during the D-phase.Figures S4–S10 present these organized examples from the real-world subset.
A.5 More Qualitative Results … A.9 Failure Analysis
The supplementary sections provide qualitative visualizations, controlled metric diagnostics, a detailed VQA pipeline, and a case study of recurring LingBot-World failures. Together, they examine evaluation robustness, phase-specific degradation, conditioning effects, and distinct failure patterns across synthetic and real-world clips.
- A.5 More Qualitative Results; A.6 Additional Radar Visualizations: Additional figures compare camera trajectories and sampled V-D-R frames across synthetic and real-world clips, while radar plots summarize complementary VQA perspectives.The qualitative comparisons include LingBot, Wan2.2, LTX-Video, FantasyWorld, and SVC; the radar plots provide high-level and fine-grained summaries.
- A.7 Ablation Studies: ORS remains stable under coverage-threshold perturbations, with mean variation of 0.013 for LingBot-World and 0.020 for StableVirtualCamera across eight configurations.The sweep uses 30 clips and threshold ranges that exclude tiny masks and overly large background masks.
- A.7 Ablation Studies: Semantically equivalent ORS prompts produce nearly identical scores, whereas truncated prompts degrade substantially, indicating semantic identity sensitivity rather than low-level pattern matching.The prompt sweep tests five formulations: the original subject phrase, four rephrasings, and a short first-word-only prompt.
- A.7 Ablation Studies: LingBot-World’s ORS is higher for larger objects, while StableVirtualCamera reaches near-zero ORS of 0.005 after reappearance following rotations above 120°.For LingBot-World, ORS is 0.67 for large, 0.45 for medium, and 0.31 for small objects.
- Motion-Gated Evaluation: Under ≥90° camera rotation, StableVirtualCamera reaches 92.43 Camera Controllability but only 0.012 ORS, whereas LingBot-World balances CamCtrl 75.04 and ORS 0.281.Joint reporting under controlled motion exposes a camera-tracking versus object-permanence trade-off hidden by aggregate evaluation; TI2V scores remain stable across thresholds.
- Per-Phase Fidelity Breakdown: All eight models show R-phase fidelity degradation: LingBot-World has the largest PSNR drop of ∆= 5.24 dB, while Matrix-Game2 has ∆SSIM = 0.239 and ∆LPIPS = 0.302.The phase breakdown separates Visible and Reappear fidelity, revealing differences masked by aggregate scores.
- Metric Sensitivity Analysis: Metric rankings are robust to hyperparameter changes: RAFT Motion Smoothness has τ = 1.000, DINOv2 Object Identity Consistency maintains τ ≥0.910, and Depth Anything V2 Geo3D Consistency maintains τ ≥0.860.The correlations are computed against default configurations over 200 clip–model pairs.
- Initial-State Conditioning vs. Backbone Capacity: Providing the V-frame improves Wan2.2 GT-aligned fidelity more than scaling from 5B to 14B, while the 14B model without it achieves the highest self-consistency metrics despite lower fidelity.Adding the V-frame improves PSNR by 4.2 dB for 5B and 4.7 dB for 14B, and reduces LPIPS by 0.20 and 0.16, respectively.