PaperlayerView exact revision on arXiv

Source-linked AI summary

Execution Before Memory: Diagnosing Memory Failures in Video World Models

Weiyang Jin, Jiwen Yu, Sihan Xu, Yuwei Niu, Xihui Liu

arXiv:2610.23542v1cs.CVcs.CL

TL;DR

Memory tests for video world models combine scene setup, execution of a requested change, and the visible result, leaving scores unable to distinguish these stages. MIST separates them with a delayed reveal, and M3-Agent revises instructions to make the change occur. M3-Agent improves reveal and benchmark scores, while the evidence does not establish that the model stored the target state.

  • Problem

    Existing tests combine whether a scene was set, a requested change was carried out, and the result appears in the last frame, making a low score ambiguous.

    Source: §1 · pp. 1–2

  • Method

    MIST separates scene setup, leakage, and reveal; M3-Agent recursively revises instruction details to carry out the requested change.

    Source: §1 · p. 3, §3 · p. 5

  • Results

    MIST reveal rises from 0.147 to 0.492; WRBench T rises 26.1% (0.452→0.570, n=40) and MBench-T U rises 52.7% (0.292→0.446, n=30).

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The results show what the frozen model can do once the missing move is stated, not what target state it stored.

    Source: §9 · p. 12

    The results do not establish that the backbone stored the target state; that claim would require a target-blind selector, held-out cases, and human labels on the exposed state.

    Source: §8 · p. 12

Abstract

from arXiv · show

A memory test for a video world model should say whether the scene was set, whether the asked change was carried out, and whether the last frame shows the result. Current tests fold these into one number, so a low score is easy to read as forgetting. When the change is not carried out, the hidden state transition cannot appear, so the test is not really a test of memory. We introduce MIST, an evaluation that separates the change from the memory it should leave behind. The model first sets the scene while the change stays hidden, and only later is asked to show it, with leakage before that point kept under control. This separation locates the bottleneck: the model sets the scene but does not carry out the instructed change. The same gap appears in general memory evaluations. On that basis we propose M3-Agent (Make the Missing Move), which recursively revises the details of the input instruction so that the model successfully carries out the requested instruction. Relative to the baseline, it more than triples the MIST reveal score. On the benchmarks, WRBench's score T rises by 26.1% and MBench-T's score U by 52.7%. The method meets the problem from the execution side: it shows what the frozen model can do once the move is stated, and it sets a stricter baseline.

1. Introduction

Existing memory tests combine scene setup, execution of the requested change, and the final visible state, so a low score does not identify which stage failed. MIST separates these checks, and M3-Agent revises instructions to address execution failures.

1. Introduction

  • Current tests collapse scene setup, the requested change, and whether the final frame shows its result into one score.A low score therefore does not distinguish forgetting from a change that was never carried out.Source: §1 · pp. 1–2
  • MIST scores scene setup, leakage, and final reveal separately, while M3-Agent reads the last frame and revises the failed move on the same frozen generator.Figure 1 contrasts three settings and shows their scored outcomes.Source: Figure 1 · p. 2, §1 · p. 3
  • MIST first establishes the scene while the change remains hidden, then requests the reveal while controlling leakage beforehand.Its three 5-second shots and 43 cases on Wan TI2V-5B separate scene setup from delayed state reveal.Source: §1 · p. 2
  • MIST reveals an execution bottleneck: 41 of 43 baseline clips never show the hidden change despite usually setting the scene without leakage.The last frame can still show the wrong state after revision.Source: §1 · p. 2
  • M3-Agent recursively revises instruction details; MIST reveal rises from 0.147 to 0.492, while WRBench T rises 26.1% and MBench-T U rises 52.7%.The method adds or trains no memory module.Source: §1 · pp. 2–3
  • The same missing-change pattern appears in general memory evaluations, including WRBench and MBench-T.Figure 2 depicts a child who leaves before sitting, intact candy after the camera returns, and revised MIST clips that open onto hidden objects.Source: §1 · p. 2, Figure 2 · p. 3
  • The method shows what the frozen model can do once the move is stated, but a judged last frame does not show that the target state was stored.Source: §1 · p. 3

2. Related Work

Prior work studies video generators as world models, hidden-state evaluation, and test-time refinement, but the cited lines do not establish that a hidden state was kept and updated. MIST applies delayed reveal to stage-resolved evaluation, while M3-Agent revises instructions to make the requested change occur.

Video generators as world models

  • Long-horizon world-model systems add packing, retrieval, or persistent state to reduce drift, but the cited lines do not show that hidden state was kept and updated.Source: §2 · p. 3

Judges and test-time refinement

  • Figure 3 illustrates MIST's separate reveal factors and an M3-Agent revision that changes the tin's contents to an orange marble.It identifies Wan TI2V-5B, Wan T2V-A14B, and MiniMax-H3 among the paper's generators.Source: Figure 3 · p. 4

Hidden state and stage-resolved evaluation

  • StEvo-Bench, MemoBench, and the Video Shell Game study occluded updates or unseen swaps; WRBench and MBench add a camera departure and later check.Temporal and physical suites also separate transition occurrence from endpoint correctness.Source: §2 · p. 4
  • MIST applies delayed reveal to this stage-resolved principle without replacing Official WRBench, Official MBench, or the Video Shell Game.Source: §2 · p. 4

Judges and test-time refinement

  • VideoRepair, PhyT2V, Video-T1, and VQQA diagnose or rewrite videos for quality, physics, or preference, whereas M3-Agent revises the instruction so the model carries out the requested change.The paper claims no numbered comparison with those tools.Source: §2 · p. 4

3. MIST: separating scene setup from delayed state reveal

MIST separates scene setup from delayed reveal so a missing requested change is distinguishable from an incorrect final state. It measures whether the move occurs, the last frame is readable, and that frame matches the target.

3. MIST: separating scene setup from delayed state reveal

  • Each case uses three 5-second shots: scene setup, an opaque gap, then a requested reveal, with the first two shots locked across reveal comparisons.The baseline prompt and gap omit the answer; target access in ablations is marked and audited.Source: §3 · pp. 4–5
  • MIST distinguishes whether the required move occurs, whether the last frame is readable, and whether it matches the target.The clip comes from a frozen generator, and a fixed VLM judge evaluates it.Source: §3 · p. 5
  • The evaluation reports setup, gap leakage, reveal, and inside separately because a total cannot distinguish a missing action from an incorrect state.A valid setup followed by a closed lid is a reveal-path failure, not evidence of forgetting.Source: §3 · p. 5
  • WRBench and MBench-T use unique scores that give no credit when a required step is missing; their path requires the requested event, not only a final match.MIST varies target states and the operations needed to reveal them.Source: §3 · p. 5, Table 1 · p. 5

4. Pre-experiments: the target is named, but the reveal is missing

The pre-experiments locate baseline failure at the reveal/action stage: setup is usually successful, and the hidden gap does not leak the answer. Revising the failed move improves reveal, but the target must also be named for the strongest best-of-five result.

4. Pre-experiments: the target is named, but the reveal is missing

  • In 38 of 43 cases, setup is at least 0.8 while baseline reveal remains below 0.3; the missing step is the later move showing the requested state.Of 43 baseline clips, 41 never open, one opens to an empty or blocked view, and one is correct.Source: §4 · pp. 5–6
  • A packed 15-second MiniMax-H3 take shifts failures from never opening to opening onto the wrong state, but the asked change remains uncarried out.The same judge finds 3 of 43 never-open and 24 of 43 opened onto the wrong state.Source: §4 · pp. 6–7
  • Paired gap leakage is 0/43 for both baseline and rewrite, placing the reveal failure at the action gate rather than after early answer exposure.The broader 51-row export is not the paired denominator.Source: §4 · p. 6
  • A targeted rewrite raises one-kept-clip reveal by a mean paired 0.34 (95% CI 0.24 to 0.45), while setup stays flat.Reveal rises in 35 of 43 cases, is unchanged in 6, and falls in 2; more exposed states do not guarantee correct states.Source: §4 · p. 6, Table 2 · p. 6
  • Reveal improves across presence/count (0.132 to 0.549), binding (0.147 to 0.464), and history-update (0.163 to 0.454) operations, while setup stays flat.The rewrite must state both the move and the evidence expected in the last frame.Source: §4 · p. 6
  • The diagnosis-guided rewrite has the strongest best-of-five reveal and inside performance; the reported comparison is 0.763 versus 0.684 for answer-only. Without the target, it reaches 0.174 and leaves 28 of 43 lids closed.On the matched six-case subset, the same-prompt running best reaches 0.580, while the rewritten move reaches 0.884.Source: §4 · p. 7

5. M3-Agent: revise the instruction for the missing change

M3-Agent reads the last frame and revises the instruction for the failed change, then regenerates with the same frozen model. Its workflow preserves the requested task details while changing the failed operation and its final-frame evidence clause.

5. M3-Agent: revise the instruction for the missing change

  • M3-Agent revises the failed instruction at test time and generates again with the same model, without adding or training a memory module.Its three steps are task recording, last-frame diagnosis, and rewriting the failed operation.Source: §5 · p. 8
  • The rewrite preserves actor, object, target state, and temporal order, changing only the failed operation and its final-frame evidence clause.Diagnosis inspects the rendered last frame once, not the generator's hidden states.Source: §5 · p. 8
  • M3-Agent keeps the highest-scoring clip among K candidates while retaining the original; selection uses the reporting scorer, making keep-best results an oracle coverage estimate rather than a deployable critic.A single revision can worsen a case, so the original remains eligible.Source: §5 · pp. 8–9
  • On MIST, the agent preserves the first two 5-second shots and rewrites only the final reveal shot, generating five candidates per arm.On WRBench and MBench-T it uses 10-second clips from frozen T2V-A14B and each suite's official camera clause.Source: §5 · p. 9

6. Evaluation setup

The evaluation keeps scorer outputs and protocols separate across its internal and official settings.

6. Evaluation setup

  • Results are labeled Official when scored by a suite's own scorer and Internal when scored by the fixed judge; adapted protocols are not pooled with these rows.Source: §6 · p. 9
  • Main claims use Wan TI2V-5B for Internal MIST and Wan T2V-A14B for Official WRBench and MBench-T.MiniMax-H3 is an Internal qualitative pilot.Source: §6 · p. 9
  • The three-chunk MBench condition is omitted, and clip lengths, seeds, judges, and budgets are listed in Tables 7 to 10.Source: §6 · p. 9

7. Main results: the same diagnosis on official scores

On both official suites, the diagnosis points to missing execution before persistence can be tested: WRBench often misses the departure or return, while MBench-T often returns to an unchanged object. M3-Agent improves the official scores, but the gains do not establish that the model stored the target state.

7. Main results: the same diagnosis on official scores

  • WRBench failures often skip the departure or return, whereas MBench-T failures return to an object that remains intact.The search must find a missing path on WRBench, but on MBench-T preserve the original clip when a fixed rewrite harms the aftermath.Source: §7 · pp. 9–11
  • WRBench’s score T rises by 26.1%, and MBench-T’s score U rises by 52.7% under keep-best selection.Table 5 reports the WRBench result; Table 4 reports the MBench-T result.Source: Table 5 · p. 10, Table 4 · p. 10, Figure 5 · p. 11
  • A readable frame is not necessarily the executed path: the best-looking WRBench clip and the clip kept for T differ on 24 of 40 items.A text-only reader can name the target, but acceptance tracks the locked last frame; agreement with official gates is much lower.Source: Table 5 · p. 10, §7 · p. 11
  • On WRBench, the score gain comes from completing the return rather than improving visual quality.The component audit attributes about 3–4% of ΔT to visual quality, 55–56% to departure, and 41% to return; a second instruction set can lower the total before the final step restores it to 0.570.Source: §7 · p. 10, Table 5 · p. 10
  • On MBench-T, keep-best can reject harmful rewrites; one clip per item scores 0.443, while a last-frame lock alone gives U = 0.228.Selection keeps the baseline on 23 items, the leave–return instruction on 5, and the last-frame lock on 2.Source: §7 · p. 10, Table 4 · p. 10

8. Discussion: what execution evidence can establish

Execution evidence can show that a model carries out a requested change, but it does not establish that the target state was stored. The evidence is bounded by the selection procedure and study scope.

8. Discussion: what execution evidence can establish

  • A high reveal is not a memory claim: if the asked change never occurs, persistence remains untested.This applies whether the clip stays closed or opens onto the wrong state.Source: §8 · p. 12
  • Keep-best is an oracle over a small candidate pool scored by the same check that reports the result.Rewrites may name the target, a target-withheld control fails, and the original remains eligible because rewriting can damage a correct case.Source: §8 · p. 12
  • The main study uses one generator family; packed MiniMax-H3 cells remain Internal, and the held-out judge check is not human validation.These boundaries constrain the study's scope.Source: §8 · p. 12
  • Claiming the backbone stored the target state would require a target-blind selector that beats baseline, held-out cases, and human labels on the exposed state.An opened lid shows that the state can be judged, not that it was remembered.Source: §8 · p. 12

9. Conclusion

MIST separates whether the scene was set, the requested change was carried out, and the last frame shows the result. M3-Agent revises the instruction, improving reveal and official scores, but these results do not establish stored memory.

9. Conclusion

  • MIST separates scene setup, execution of the asked change, and whether the last frame shows the result.On Internal cases, setup and the hidden gap usually hold, while the baseline often fails to show the change.Source: §9 · p. 12
  • M3-Agent revises the instruction so the same frozen model carries out the change, improving Internal MIST reveal and Official unique scores.The Official unique scores show opposite keep-best consequences on the two suites.Source: §9 · p. 12
  • The results do not establish that the backbone stored the target state; memory evaluations should publish the three checks separately.Source: §9 · p. 12

A. Protocol cards and budgets

This section distinguishes the settings, scorers, and budgets behind reported results, then maps related work by the questions it addresses. It cautions against treating different protocols or score components as interchangeable evidence.

A. Protocol cards and budgets

  • MIST, WRBench, and MBench-T use different generators, durations, judges, and scoring protocols; Table 7 defines WRBench 𝑇 and MBench-T 𝑈 and notes that missing returns or untriggered clips score zero.MIST uses Wan2.2-TI2V-5B with three 5-second shots and a locked 122B judge; official WRBench and MBench-T use Wan2.2-T2V-A14B over 10 seconds with released scorers.Source: Table 7 · p. 20, A. Protocol cards and budgets · p. 20
  • Table 9’s per-pass means are non-monotone, and its “inside of chosen” score is not the kept-clip inside match of 0.442.The complete running-best budget is distinguished from the kept-clip result.Source: A. Protocol cards and budgets · p. 21, Table 9 · p. 21
  • WRBench search-budget ceilings are not the reported method, adding a second skill is not monotone, and MBench-T trigger rates must be kept separate from conditional scores.The conditional scores are not MIST metrics.Source: A. Protocol cards and budgets · p. 21, Table 10 · p. 21, Table 11 · p. 22
  • When selectors cannot see the scorer, none of the scorer-free selectors separates from baseline; Figure 6 panels describe score composition, not independent causal correlations.The plotted components are contained in the official totals.Source: A. Protocol cards and budgets · p. 22, Table 12 · p. 22, Figure 6 · p. 22
  • Of 41 baseline never-opens, rewriting leaves 19 closed, opens nine to a wrong state, and makes 13 correct; the exclusive never-open count is 20 after a baseline-correct case regresses.Kept-clip reveal is 0.492, not the best-of-five 0.763; the first pass revises 33 of 43 prompts.Source: A. Protocol cards and budgets · p. 22

B. Extended literature map and positioning

C. Other generations of the same cards

The same cards behave differently in packed H3 takes and the Wan stitch, and the internal pilots and rescoring remain distinct from official benchmark results. These comparisons are bounded by their settings and sample sizes.

C. Other generations of the same cards

  • In the packed H3 43-case run, 40 clips open and 24 expose the wrong state, unlike the stitch’s 41/43 never-open rate; the packed-cell result is Internal, not Official 𝑇 or 𝑈.The packed run uses a different generator and is not the population of Table 2.Source: C. Other generations of the same cards · pp. 24–25
  • M3 raises the packed H3 continuity pass rate but is not monotone: 12 cases fail raw and pass M3, seven pass raw and fail M3, and both arms pass 19 of 43.Continuity measures hard cuts using adjacent-frame gray MAE ≥22; it is not a memory score.Source: C. Other generations of the same cards · p. 25, Table 16 · p. 26
  • The six-card Internal transfer pilot does not support a model-scale or official-protocol claim; its generators and judge differ, and the arms are mixed rather than monotone.The planned 12-case H3 expansion was paused before scoring, and Lingbot M3 arms were not run.Source: C. Other generations of the same cards · p. 25
  • Internal rescores use different scales and do not replace official scores: WRBench’s Internal reveal is 0.372 →0.668, while Official MBench-T 𝑈 remains 0.446.The Internal unique score is for a 12-case MBench-T slice and is not added to 𝑈.Source: C. Other generations of the same cards · pp. 25–26
  • A proposed 12-item TI2V-5B/A14B image-to-video protocol failed before producing clips, so it has no official score; adapted MemoBench and SANA-WM smoke tests were also excluded.The run produced 0/12 clips after an FSDP no_sync failure.Source: C. Other generations of the same cards · p. 26

D. Held-out last-frame judge

The held-out 8B reread evaluates the locked last frames without replacing the 122B judge or official scores. The accompanying qualitative evidence and transfer pilot are internal and have distinct scope.

D. Held-out last-frame judge

  • The held-out Qwen3-VL-8B reread answers YES or NO to 𝑠★𝑇 on the same locked last frames, while official 𝑇 and 𝑈 remain unchanged.Its YES is not an official 𝐷5 or 𝐷6 decision or an MBench state score; the rubrics ask different questions.Source: D. Held-out last-frame judge · p. 26, Table 20 · p. 26
  • Agreement with the 122B inside check is high on baseline frames under class imbalance, but agreement on rewritten-frame action and with official WRBench and MBench gates is lower.The main-text last-frame rate of 0.023 is a strict YES rate, distinct from mean inside of 0.047.Source: D. Held-out last-frame judge · p. 26, Table 20 · p. 26
  • Figure 7 compares H3 raw and M3 on the same cards; its leftover-fail panel shows both arms breaking continuity and showing the wrong object, while seed-61001 43-case pairs are separate.Source: Figure 7 · p. 28

Related papers

Loading 2610.23542v1…