WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models

1 INTRODUCTION

• WorldMemBench covers 260 cases across 23 categories, including 200 static-memory cases and 60 dynamic-memory cases.Static cases cover humans, objects, and environments; dynamic cases cover seven types of off-screen state transitions.Source: §1 · p. 2
• The benchmark uses fixed initial references for static revisits and event-specific checklists to assess progression and consequences against expected next states.Dynamic memory is assessed through coherent state changes rather than similarity to an unchanged initial frame.Source: §1 · p. 2
• HY-WorldPlay leads all seven static metrics but initiates only 31.7% of prescribed events, showing that strong static retention need not imply reliable dynamic evolution.Across nine models, repeated revisits expose cumulative degradation, and switching rotation axes adds a further 0.152 drop in Environment Visible.Source: §1 · p. 2

W ASD T

• The benchmark comparison table marks coverage as explicit, partial, or unreported across world-model paradigms and evaluation capabilities.Its legend also distinguishes action-driven and language-driven model inputs.Source: Table 1 · p. 3
• Human evaluation shows 78.5%–92.5% pairwise agreement across six static metrics and Spearman ρ = 0.732 for Dynamic Memory Overall.No evaluated model leads all four dimensions, and context-access comparisons hold the model and generation settings fixed.Source: W ASD T · p. 3

2.1 VIDEO WORLD MODELS

• Approaches to spatial memory include long-context modeling, historical-context reconstruction, and field-of-view-based frame retrieval for preserving previously observed scene content.These approaches accompany progress in long-horizon interactive video generation and camera and action control.Source: §2.1 · p. 3

2.2 EVALUATION OF VIDEO WORLD MODELS

• Video-generation and world-model benchmarks assess quality, controllability, interaction, and physical plausibility, but these dimensions alone do not establish reliable retention and updating of world information over time.Memory-oriented benchmarks address interrupted-observation state evolution and multi-revisit scene recall.Source: §2.2 · pp. 3–4
• WorldMemBench combines Human, Object, and Environment memory with dynamic event evolution, camera following, and video quality, and separates event initiation from continuation when scoring dynamic memory.The accompanying figure presents the case taxonomy, evaluation counts, and examples of targets, depth, and event states around revisits.Source: §2.2 · p. 4, Figure 2 · p. 4

3.1 EVALUATION DIMENSIONS

• WorldMemBench evaluates static retention of previously observed humans, objects, and environments, and dynamic event progression and persistence of consequences while content is out of view.Each dynamic case specifies an initial state, an action, and a next state; quality and camera-following dimensions provide complementary diagnostics.Source: §3.1 · p. 4

3.2 DATA COMPOSITION

• The benchmark contains 260 core cases: 60 Human, 70 Object, 70 Environment, and 60 Dynamic Memory cases.The first three families form 200 static-memory cases; video quality is evaluated on static rollouts, while camera following reuses 85 suitable static cases.Source: §3.2 · p. 4

3 WORLDMEMBENCH CONSTRUCTION

• The 23 content categories span entity types, scene configurations, and seven dynamic state-transition mechanisms.The benchmark’s construction probes whether observed content and evolving states remain consistent after periods outside the camera’s field of view.Source: §3 · p. 4, §3.2 · p. 5

3.3 CAMERA TRAJECTORIES

• Translation trajectories preserve viewing direction around a square loop, while rotation trajectories keep camera position fixed and turn away and back through yaw or pitch.The benchmark includes repeated same-axis returns and LRUD returns that switch axes; V denotes initial observation, Di a departure, and Ri its prescribed revisit.Source: §3.3 · p. 5, Figure 3 · p. 5

4.1 MULTI-REVISIT PROTOCOL

• Each prescribed revisit is compared with a fixed initial reference, and rollout scores average first across revisits and then across targets.All prescribed revisits count, including those without valid observations.Source: §4.1 · p. 5

4.2 GENERAL VIDEO QUALITY

• Video quality samples every fourth frame and reports normalized imaging and aesthetic scores separately, averaged across frames and then cases.Imaging Quality uses MUSIQ-SPAQ; Aesthetic Quality uses the LAION aesthetic predictor based on CLIP features.Source: §4.2 · p. 6

4.3 CAMERA FOLLOWING

• Camera Following reports translation and rotation relative pose errors to measure adherence to prescribed camera motion and help identify viewpoint errors.Translation trajectories are independently aligned using Sim(3); rotation uses DA3 orientations without Sim(3) alignment. Both metrics are dimensionless, with lower values better.Source: §4.3 · p. 6

4.4 STATIC MEMORY

• Static memory scores human identity and appearance, object appearance and geometry, and environmental visibility, depth ordering, and 3D reconstruction against initial references.Human and object observations are compared with fixed initial references; object geometry normalizes silhouettes to remove translation and uniform scale while preserving shape changes. Environment metrics are reported separately.Source: §4.4 · pp. 6–7

4.5 DYNAMIC MEMORY

• Dynamic memory checks event onset and scores eight fixed-answer questions on progression, persistence or reset, final state, and causal or physical consistency.If the event does not start during the initial hold, all eight question scores are zero; otherwise, each matching answer earns one point.Source: §4.5 · pp. 7–8

5.1 SETUP

• The evaluation covers nine models under a unified protocol, with each configured according to the defaults in its official release.Model-specific resolutions, frame rates, temporal generation strategies, control adaptations, and inference settings are detailed in Appendix C.Source: §5.1 · p. 8

5.2 EVALUATION OF WORLD MODELS

• HY-WorldPlay leads all seven Static Memory metrics, but LingBot-World and YUME score higher on Dynamic Memory; no model leads across all four dimensions.Qualitative examples show HY-WorldPlay most consistently preserving identity, object geometry, and scene layout across three revisits; human-study results support alignment between automatic metrics and human judgments.Source: §5.2 · p. 8

5.3 MULTI-REVISIT ANALYSIS

• From R1 to R3, nine-model averages fall by 0.229 in Object Appearance, 0.217 in Human Identity, and 0.205 in Object Geometry.Environment Reconstruction Score decreases by 0.196; degradation remains on pose-qualified returns, where Visible, Depth, and Reconstruction scores fall by 0.152, 0.135, and 0.139.Source: §5.3 · pp. 8–9

5.4 CROSS-AXIS REVISIT ROBUSTNESS

• Across nine models, cross-axis LRUD returns degrade more than same-axis LRLR returns in 50 of 63 model–metric pairs; Environment Visible has the largest added drop, 0.152.The comparison defines degradation as R1 minus R2 and contrasts LRUD with LRLR; the figure tracks per-model changes across three LRLR revisits.Source: §5.4 · pp. 9–10, Figure 5 · p. 10

5.5 DIAGNOSTIC CASE STUDY: TRAINING-FREE CONTEXT ACCESS

• FoV achieves the best scores on eight of eleven capability metrics and outperforms Sliding Windows on all eleven, while FullKV retains advantages on three static metrics.Across 660 matched videos, mean latency is 262.7 s for FoV and 260.4 s for Sliding Windows, versus 322.2 s for FullKV.Source: §5.5 · p. 10, Table 3 · p. 10

6 CONCLUSION

• Across nine models, no model leads across all dimensions, strong static retention does not ensure reliable dynamic evolution, and repeated or cross-axis revisits reveal further degradation or sensitivity.Human studies support alignment between automatic assessments and human judgments; a controlled context-access study also compares memory performance and inference efficiency.Source: §6 · pp. 10–11

A OVERVIEW OF THE APPENDIX

• The appendix covers model inference settings, Dynamic Memory assessment, qualitative comparisons, human evaluation, revisit analyses, context-access policies, and benchmark limitations.It also begins with a statement on LLM usage during manuscript preparation.Source: A OVERVIEW OF THE APPENDIX · p. 15

• All formal inference runs use frozen random seeds and configurations, so settings do not vary across cases or trajectories.Runs use machines equipped with eight NVIDIA H20-3e GPUs.Source: C INFERENCE SETTINGS · p. 15
• HY-WorldPlay conditions generation on per-frame camera-to-world poses, while Matrix-Game 3 translates trajectories into keyboard and mouse controls.HY-WorldPlay uses seed 1 and four-step distilled autoregressive sampling; Matrix-Game 3 uses 12 iterations and generates 497 frames in total.Source: C INFERENCE SETTINGS · p. 15
• Matrix-Game 3.5 builds PRoPE conditions from camera intrinsics and poses, while SANA-WM uses chunk-level Plücker conditions in a streaming generation pipeline.Matrix-Game 3.5 uses six blocks for a continuous 505-frame rollout; SANA-WM uses four-step self-forcing sampling and outputs 481 frames.Source: C INFERENCE SETTINGS · p. 15
• YUME encodes trajectories as segmented camera captions rather than numerical camera poses; LingBot World interpolates poses and converts them to framewise relative poses and Plücker conditions.YUME produces 493 frames for LRLR and LRUD, and 551 for LRLRLR; LingBot World uses eight-card causal autoregressive inference with FullKV attention.Source: C INFERENCE SETTINGS · p. 15
• LingBot World v2 uses the LingBot World pose and chunk protocol; FoV and Sliding Windows are reserved for the controlled context-mechanism comparison.The main nine-model results use seed 1 and FullKV attention; the context comparison freezes other conditions, and Table 4 describes formal artifact specifications and sampling configurations.Source: C INFERENCE SETTINGS · pp. 15–16, Table 4 · p. 16
• Evoke uses per-case intrinsics and a 501-frame absolute camera-to-world trajectory, while DreamX World uses chunk-relative PRoPE conditions and retains the full generation history.Evoke keeps 21 static pose samples for the final return view; DreamX World decodes 126 latent frames into 501 pixel frames.Source: C INFERENCE SETTINGS · p. 16

D.1 ASSESSMENT PROTOCOL

• Each rollout is sampled every eight frames, including its first and last frames, and presented chronologically with the checklist and planned camera-phase ranges.The evaluator first judges whether the central event starts during the initial hold, then answers eight questions; if the event has not started, all checklist scores are zero.Source: §D.1 · p. 16

D.2 EVENT INITIATION AND CONDITIONAL DYNAMIC SCORES

• The reported gated score equals the event-start rate multiplied by the conditional checklist score over started events.HY-WorldPlay starts 19 of 60 events and scores 0.928 conditionally, yielding 0.294 overall; LingBot World starts 56 of 60 and scores 0.866 conditionally, yielding 0.808 overall.Source: §D.2 · p. 16, Table 5 · p. 17

D.3 ASSESSMENT PROMPT

• The prompt requires visible before-and-after evidence for change claims and at least two initial-hold images showing concrete progress to count an event as started.Judges see chronologically sampled images without the target generation prompt or expected answers, and must not infer unseen events or states.Source: §D.3 · p. 17

E QUALITATIVE RESULTS

• HY-WorldPlay preserves identity, appearance, wardrobe geometry, room content, and spatial structure through R3, matching its lead across all seven Static Memory metrics.Evoke remains comparatively stable, while other models show identity drift, target disappearance, or scene-layout changes at later revisits.Source: E QUALITATIVE RESULTS · p. 18
• Human and object cases provide qualitative comparisons of video world models, complementing the Environment case in Fig. 4.The comparisons highlight target regions with red boxes and show metric scores for each revisit.Source: E QUALITATIVE RESULTS · p. 18, Figure 6 · p. 19, Figure 7 · p. 20
• Evoke is comparatively stable in the Human and Object examples, consistent with its second-best results on the four entity-level metrics.Other models show identity drift, target disappearance, or scene-layout changes at later revisits.Source: E QUALITATIVE RESULTS · p. 18

F HUMAN EVALUATION

• 504 blinded pairs across six Static Memory metrics were each judged by three annotators, and the automatic preference was compared with decisive human majority votes.Majority ties and three-way A/Tie/B splits are excluded from the pairwise-agreement denominator; model-level Spearman ρ uses all raw votes on the same 84 tasks.Source: F HUMAN EVALUATION · p. 21
• 78.5%–92.5% pairwise agreement with human majority votes is reached by all six Static Memory metrics.At model-ranking level, Environment Visible, Human Appearance, and Human Identity attain Spearman ρ = 0.988, 0.952, and 0.929, respectively.Source: Table 6 · p. 21, F HUMAN EVALUATION · p. 21
• Dynamic Memory evaluation reaches Overall Spearman ρ = 0.732 and PLCC = 0.808 against human judgments, with highest sub-dimension agreement on Return Progress.Return Progress attains ρ = 0.684 and PLCC = 0.683.Source: Table 7 · p. 22, F HUMAN EVALUATION · p. 22

G TRAJECTORY-WISE AND CROSS-AXIS RESULTS

• HY-WorldPlay leads all seven Static Memory metrics on LRLR and LRLRLR, and six on LRUD, while YUME leads Imaging across all three trajectories.Evoke leads Human Appearance on LRUD; the results show that strong visual quality does not necessarily coincide with strongest memory retention.Source: G TRAJECTORY-WISE AND CROSS-AXIS RESULTS · p. 22, Table 8 · p. 23
• +0.152 is the largest mean excess degradation under axis switching, for Environment Visible; Object Appearance and Object Geometry follow at +0.127 and +0.120.Both LRLR and LRUD show lower nine-model mean scores at the second revisit across all seven metrics, with larger drops under LRUD.Source: G TRAJECTORY-WISE AND CROSS-AXIS RESULTS · p. 22, Table 9 · p. 23
• All nine models show positive cross-axis excess degradation on Environment Visible, but LingBot World’s excess is +0.368 there versus Evoke’s +0.002.For Environment Depth, LingBot World’s excess is +0.465 while Evoke’s is −0.098, illustrating substantial model-level variation.Source: G TRAJECTORY-WISE AND CROSS-AXIS RESULTS · p. 23, Table 10 · p. 24

G.1 MEMORY EVALUATION UNDER DEPARTURE AND RETURN CONTROLS

• For Human and Object, qualifying returns require tracker-supported departure and valid post-return observations for every initially evaluated target; Environment returns require departure of at least 30° and return within 30° of the initial orientation.Within-model comparisons use fixed case sets across the compared revisits, and retained sample sizes are reported because some model subsets contain only one or two cases.Source: §G.1 · p. 24, Table 11 · p. 25
• Across 431 qualifying Environment model–case pairs, Visible, Depth, and Reconstruction decrease from R1 to R3 by 0.152, 0.135, and 0.139, respectively.These decreases are smaller than the unfiltered values of 0.161, 0.138, and 0.196, but remain positive; tracking-supported subsets also show mean decreases across all four entity metrics.Source: §G.1 · pp. 24–25, Table 12 · p. 25
• On 281 qualifying case pairs, mean additional degradation under LRUD is 0.128 for Environment Visible, 0.076 for Depth, and 0.056 for Reconstruction.All four entity metrics retain a positive mean cross-axis penalty, although Object Appearance has four positive and four negative model-level effects.Source: Table 12 · p. 25, §G.1 · p. 25
• Mean effects remain positive for all three Environment metrics under 15° and 5° return tolerances, but the tighter criteria reduce coverage and change the evaluated samples.The 5° cross-axis comparison contains only 30 case pairs from four models.Source: §G.1 · pp. 25–26, Table 13 · p. 26
• Residual camera-position, occlusion, and estimation differences may affect scores, so the analysis supports the degradation trends without attributing the remaining decrease exclusively to forgetting.The benchmark also notes that effect sizes across tolerances reflect changes in evaluated samples.Source: §G.1 · p. 26

H ANALYSIS OF CONTEXT-ACCESS POLICIES

• The controlled comparison uses LingBot World v2 without retraining, fixes the checkpoint and generation conditions, and varies only the temporal KV-access policy.FullKV accesses the entire history; the 20-latent Sliding Windows and FoV budgets differ in their allocation, with FoV including retrieved history selected by field-of-view overlap.Source: H ANALYSIS OF CONTEXT-ACCESS POLICIES · p. 26
• The policy comparison uses matched evaluation sets of 660 videos per policy, and latency is measured for generating a 493-frame video.The reported speedup is FullKV’s mean latency divided by the corresponding policy’s mean latency.Source: H ANALYSIS OF CONTEXT-ACCESS POLICIES · p. 26
• FoV raises target-viewpoint returns from 57.6% to 65.3% and lowers mean return rotation error from 31.0° to 26.7° relative to FullKV across 490 Environment revisit slots.These gains accompany improvements in Human Identity, Human Appearance, Environment Depth, and Environment Reconstruction.Source: H ANALYSIS OF CONTEXT-ACCESS POLICIES · p. 26, Figure 10 · p. 27
• FoV improves all four Dynamic Memory sub-metrics over both FullKV and Sliding Windows, whereas Sliding Windows improves Final State from 0.592 to 0.600 but scores lower on Persistence/Reset and Causal/Physics than FullKV.Sliding Windows matches FullKV on Return Progress (0.667); the erasing example illustrates differences in task-relevant object consistency.Source: H ANALYSIS OF CONTEXT-ACCESS POLICIES · p. 26, Figure 10 · p. 27

I LIMITATION AND FUTURE WORK

• The benchmark focuses on controlled translation and rotation trajectories over rollouts of several hundred frames; free-form or compound traversal and jointly selected camera and object actions remain outside its scope.Dynamic Memory also relies on frozen case-specific VLM checklists, with Causal/Physics identified as the most challenging sub-dimension.Source: I LIMITATION AND FUTURE WORK · p. 27