Tracking Is Not Permanence: What Video World Models Keep of a Hidden Object

• The paper distinguishes permanence from visible-object tracking and notes that above-chance scores on occlusion benchmarks do not reveal whether a predictor keeps hidden-object state.Source: §1 · p. 1
• The belief probe compares hidden-region predictions with encoder representations of two futures that differ only inside the hidden region.It uses a two-alternative choice with chance 0.5 and a never-shown-object bias control.Source: §1 · pp. 1–2
• Under the main query, moving objects read absent within 0.3 s, while stationary objects persist in only 13–41% of scenes on average.The last-view-copy baseline reads 97–98% on average.Source: §1 · p. 2
• The encoder reads object presence at 1.00 and keeps a rigid box’s contents decodable for 3.5 s, while the predictor’s output contains the ball in 2% of scenes after the box closes.The encoder’s own probe reads the predictor’s output.Source: §1 · p. 2
• Three thousand predictor-only steps on synthetic containers install a containment prior and raise IntPhys-2019 from 84.2% to 93.3%, though a curriculum without containers also raises the benchmark score.Which training habit the benchmark credits depends on its scoring rule.Source: §1 · p. 2
• Continued training with tube masks produces moving-object carry-over on manipulation and internet-style video, while VideoMAE and Cosmos show different failures on hidden objects.Cosmos keeps a stationary hidden object but not one carried inside a moving container; V-JEPA 2 and Cosmos fail alike on that case.Source: §1 · p. 2

• Occlusion benchmarks score one answer per clip, whereas this paper reads a released predictor’s state at every step.The benchmark literature includes IntPhys, Physion, InfLevel, LA-CATER, TCOW, and the Perception Test.Source: §2 · p. 2
• The belief probe compares a frozen predictor’s inferred hidden tokens with encoder representations of two worlds differing only in the hidden region.Its P(present) readout is measured per 0.27 s tubelet against chance 0.5.Source: Figure 1 · p. 3
• ADEPT, PLATO, trackers, and slot models build permanence into their designs or train specifically for it; this paper asks whether a general predictor acquires it.Source: §2 · p. 3
• V-JEPA predicts masked tubelet features, VideoMAE reconstructs masked pixels, and Cosmos-Predict1 predicts the next discrete video token.Prior work found limited-temporal masks make V-JEPA’s task easier and cost downstream accuracy; this paper tests what its mask choice leaves of a hidden object.Source: §2 · p. 3

3 READING A BELIEF OUT OF A FROZEN PREDICTOR

• The readout compares a prediction for hidden tokens with encoder references for possible and impossible worlds that share all visible tokens.The predictor is the only route by which the object can reach the inferred hidden tokens.Source: §3 · p. 3, §3.2 · p. 4

3.1 MINIMAL PAIRS

• Rendered minimal pairs differ only inside the ball’s trajectory band after the event, with the ball continuing in one world and vanishing in the other.Scenes vary distractor count and ball radius; five seeds per cell yield 60 decisions per point.Source: §3.1 · p. 3

3.2 READOUT

• P(present) is the fraction of scenes whose prediction is nearer the present reference; carry-over is the last tubelet at or above 0.5 with its control below 0.5.Because uncertain regression predictions can move toward either reference, the section also reports signed distance and relative surprise.Source: §3.2 · p. 4

3.3 CONTROLS AND READOUT VARIANTS

• Controls include never-visible objects, static-versus-moving worlds, end-of-clip reveals, visible containers, and real-hand clips with coin-in, removed, and never-shown conditions.Source: §3.3 · p. 4

3.4 PROBES FOR OTHER PARADIGMS

• VideoMAE is tested by pixel reconstruction, while Cosmos-Predict1-4B is evaluated on the first differing token chunk after a shared absent-world prefix.The Cosmos comparison uses within-scene log-likelihood differences and controls for never-shown boxes or exits.Source: §3.4 · pp. 4–5

3.5 TRAINING INTERVENTIONS

• Training interventions vary mask type, synthetic containers, continued pretraining, and real-hand curricula; IntPhys-2019 is evaluated with fixed-context and minimum-surprise scoring rules.The interventions include predictor-only and encoder-plus-predictor training on rendered scenes, manipulation video, internet-style clips, and real-hand examples.Source: §3.5 · p. 5

4.1 RELEASED MODELS LOSE A HIDDEN MOVING OBJECT UNDER A DENSE CONTEXT

• At 0.27 s, ViT-H’s moving-object belief falls from 0.92 to 0.00; all released scales lose it after one tubelet, while stationary objects persist only in part.The projected prediction retains a trace, but remains below the midpoint between present and absent worlds from 0.27 s onward.Source: §4.1 · pp. 5–6, Figure 2 · p. 6

4.2 THE PREDICTOR INPAINTS LOCALLY UNDER A DENSE CONTEXT

• Revealing the ball in the final four frames changes only the adjacent hidden tubelet in the released predictor, whereas a trained predictor carries it across the gap and revises from future evidence.When the reveal shows the ball absent, synthetic curricula continue to predict its presence across the gap.Source: §4.2 · p. 6, Figure 3 · p. 7

4.3 THE INFORMATION IS IN THE ENCODER AND THE PREDICTOR BARELY USES IT

• Encoder tokens decode ball presence at 1.00 and closed-box contents at 0.91–1.00 for 3.5 s, but the predictor’s output probe finds the ball in only 2% of scenes at 0.53 s.With real closing hands, encoder coin decodability falls to 0.56–0.59 after closure.Source: §4.3 · pp. 6–7

4.4 THE DEFICIT IS NOT INTRINSIC TO LATENT PREDICTION

• Continued natural-video training with tube masks yields 1.1–1.6 s of moving-object carry-over, while a mask mixture freezes objects; single-mask synthetic training also yields 1.3–1.6 s of carry-over.The continued-training runs do not keep stationary objects, while the mixture keeps stationary objects and balls inside boxes in place.Source: §4.4 · p. 7, Figure 4 · p. 8

4.5 PIXEL-RECONSTRUCTION AND AUTOREGRESSIVE MODELS UNDER MATCHED PROBES

• VideoMAE-L reconstructs a hidden 32 px ball for 0.3 s, while Cosmos expects a ball in a stationary box but not one carried inside a moving box.Against the exit control, Cosmos’s moving-box memory is negative at 1.07 and 2.13 s.Source: Figure 5 · p. 8, §4.5 · p. 8

4.6 WHAT TRAINING INSTALLS

• 3,000 predictor-only steps raise container belief from at most 0.17 to 1.00; ViT-H IntPhys-2019 rises from 84.2% to 93.3%, but no-container curricula also reach 93.9–94.7%.Which training habit the benchmark rewards changes with its scoring rule, so accuracy alone does not establish permanence.Source: §4.6 · pp. 8–9, Figure 6 · p. 9

5 CONCLUSION AND LIMITATIONS

• The tested V-JEPA 2 predictors retain little about hidden objects despite encoders representing visible objects; the conclusions are limited to the tested models, renderer, curricula, and evaluation protocols.The study excludes action-conditioned V-JEPA 2 and V-JEPA 2.1, lacks a two-world test on real occlusion video, and uses thresholded decisions over 60 scenes per point.Source: §5 · p. 10

AI USE STATEMENT

• Generative AI assisted with framing, experiments, implementation, analysis, and writing, but authors reviewed the work and take responsibility for its final content.The authors state that no generative model produced images, video, or data points.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

• Natural video came from existing datasets; identifiable people appear in frames, but the authors redistribute none and report no participant-level results beyond held-out group accuracy.Coins composited onto real hands are synthetic overlays that change no person's appearance or action.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• The authors say they will release the probe, renderer, training and evaluation scripts, manifests, and job scripts that produced the reported numbers.The statement also points to appendices documenting hyperparameters, preprocessing, splits, and the IntPhys-2019 protocol.Source: REPRODUCIBILITY STATEMENT · pp. 10–11

A FULL RESULT TABLES

• The table covers five training interventions, with the IntPhys-2019 gain reported for ViT-H using two context frames.All interventions start from the released encoder; all except (v) also start from the released predictor.Source: Table 2 · p. 14
• Intervention (v) starts from a predictor already trained through (i), a container curriculum, and hand-anchored masks.Source: Table 2 · p. 14
• The table caption identifies the interventions and evaluation conditions but supplies no individual result values.Source: Table 2 · p. 14

B DATASETS AND PREPROCESSING

• Natural-video training uses 59k clips from EgoDex, EPIC-KITCHENS-100, and DROID; the 908 training events exclude participants P27 and P28.The held-out sets include 11 unseen-participant events and 19 EgoDex events.Source: B DATASETS AND PREPROCESSING · p. 14
• MagicCATs contributes 166 clips with 81 detectable closing events, while MMC contributes 36 videos with 99 human-rated surprise moments and 21 detectable closing events.Source: B DATASETS AND PREPROCESSING · pp. 14–15
• Table 3 reports that released models reach at most 0.17 belief at 0.27 s and at most 0.18 afterward for a moving hidden object.The temporally-partial-mask predictor retains a moving object for 1.1 s and a stationary one through the full window.Source: Table 3 · p. 15
• Table 4 reports stationary-object beliefs of 0.77–1.00 across steps for single-mask predictors, but containment beliefs at 0.53 s of only 0.05, 0.18, and 0.18 for causal, partial, and tube masks.The caption says IntPhys-2019 rises under all three mask kinds; these runs have no never-visible control of their own.Source: Table 4 · p. 15
• The renderer uses 256×256 scenes with a moving ball, optional distractor rectangles, and an opaque 80×88 px box.The ball radius ranges from 4 to 32 px, and the event occurs at the clip midpoint.Source: B DATASETS AND PREPROCESSING · p. 15

C TRAINING DETAILS

• Predictor-only natural-video training uses a frozen released ViT-H encoder and checkpoint-initialized predictor for 20k steps; masks are sampled per batch as causal (30%), temporally-partial (50%), and tube (20%).The loss is L1 on encoder features with a learned per-token Laplace scale. Variants use 32-frame windows throughout, reweight later slots, or use a 24-layer predictor.Source: C TRAINING DETAILS · pp. 15–16
• Continued tube-mask training for 16k steps produces moving-object carry-over on manipulation and internet-style video, while the mask mixture freezes the object.The table covers two seeds on manipulation video and an internet-style video evaluation.Source: Table 5 · p. 16
• Three thousand predictor-only steps on synthetic containers raise hidden-ball belief to 1.00 with both controls at 0.00.The table caption says the released predictor loses the ball within 0.5 s; stationary-box controls are 0.00–0.02.Source: Table 6 · p. 16
• The synthetic container curriculum runs 3,000 steps with 70% container-mask batches and 30% generic-mixture batches, covering enter, miss, exit, and none scenarios.Ablations vary the container-mask fraction and include a frozen-predictor scale-head-only condition.Source: C TRAINING DETAILS · p. 16
• Continued pretraining uses a ViT-L with an EMA target for 16k steps, while the real-hand curriculum uses ViT-H and mixes real-hand, synthetic-container, and natural-video batches.The continued-pretraining mask variants are tube-only or a mask mixture; the real-hand recipe uses 35% real-hand and 20% synthetic-container batches.Source: C TRAINING DETAILS · pp. 16–17
• The real-hand table evaluates coin-in, removed, and never worlds on participants unseen in training, with events from EgoDex, EPIC-KITCHENS, and MagicCATs.The clean recipe uses intervention (v) and two seeds; the table reports events of 8, 4, and 39 for the three datasets.Source: Table 7 · p. 17
• Cosmos-Predict1-4B is evaluated by teacher-forced log-likelihood on the reveal chunk, with memory defined as the per-scene difference between the main condition and a control.The table includes 30 scenes per row.Source: Table 8 · p. 17
• The appendix fixes grouped cross-validation folds because library-version changes shifted closed-hand accuracy by up to 0.16, and reports roughly 150 GPU-hours for main experiments.It also lists software versions, model checkpoints, and the varied learning rates, mask weights, and real-hand batch fractions.Source: C TRAINING DETAILS · pp. 17–18

D ADDITIONAL CONTROLS

• From the second tubelet onward, v2 places the hidden moving object where it was last seen; “gone” is most frequent only at 2.7–3.2 s.Its three further variants track the v2 curve within 0.12 at every point but one and share its 1.1 s carry-over.Source: Figure 7 · p. 18, D ADDITIONAL CONTROLS · p. 18
• The mask mixture keeps a ball inside moving and stationary boxes at 1.00 for every hidden duration, above the never-visible and exit controls.For the moving box, those controls are 0.03 at 0.53 s and 0.00 afterwards.Source: D ADDITIONAL CONTROLS · p. 18
• The 24-layer temporally-partial-mask variant holds the static world at 0.98 at 3.2 s, while v2’s most frequent answer is “gone” only at 2.7–3.2 s.The mixture model’s stationary belief of 1.00 is above its never-visible control from 0.53 s onwards.Source: D ADDITIONAL CONTROLS · p. 18, Figure 7 · p. 18

F AUTOREGRESSIVE READOUT

• Against the exit control, Cosmos-Predict1-4B's moving-box evidence is −2.7 [−9.0, 3.0] and −10.3 [−17.4, −3.6] nats, while stationary-box evidence is +20.7 [14.5, 27.0] and +18.7 [13.3, 24.2] nats.The authors interpret the negative moving-box values as no positive memory and do not explain the sign further.Source: F AUTOREGRESSIVE READOUT · p. 19

G REAL-HAND EVENT COUNTS AND LINEAR PROBES

• Short-closure events number 8 of 19 in EgoDex, 4 of 11 in EPIC-KITCHENS P27/P28, 39 of 81 in MagicCATs, and 11 of 19 in MMC.A removed world exists for 3, 2, 24, and 9 events, respectively.Source: G REAL-HAND EVENT COUNTS AND LINEAR… · p. 19
• On MMC's 11 events, the clean recipe s2 reads 0.82 / 0.56 / 0.36 for coin-in / removed / never.Across the listed datasets, closed-hand decodability and visible-coin controls vary by model and dataset.Source: G REAL-HAND EVENT COUNTS AND LINEAR… · p. 19

H REAL-MAGIC READOUTS

• Across MMC, MagicCATs, and FunQA, released V-JEPA 2's prediction error at human-rated surprise moments is indistinguishable from motion energy; only 3 of 99 MMC moments fall within one second of a detectable hand reopening.The evaluation spans three model scales and six readouts.Source: H REAL-MAGIC READOUTS · p. 19
• The MMC hand-region belief probe has per-clip AUC 0.49 for released ViT-H and 0.44–0.49 for trained checkpoints, against chance 0.50.Only 11 of 99 rated moments fall within two seconds of a hand reopening.Source: H REAL-MAGIC READOUTS · p. 19
• Intervals defined using the stated HIDE/REVEAL rules, without hand-annotation checks, had medians of 1.5 s (MMC; none shorter than 0.27 s) and 3.0 s (MagicCATs); MagicCATs surprise ratings did not depend on interval duration (ρ = +0.05 over 130 clips).Source: H REAL-MAGIC READOUTS · p. 20

Symmetric readouts, matched controls and curriculum ablations

• The predicted-token probe separates the main condition from controls at 0.99–1.00 across delays, indicating the predictor output is not blind to the context.The encoder’s probe on predicted tokens and copy-last-view provide checks against readout asymmetry and simple freezing of the last visible state.Source: I SYMMETRIC READOUTS, MATCHED CONTROLS… · p. 20
• From 0.53 s, the encoder-probe reads a moving ball in at most 20% of scenes and a closed-box ball in 2%, while copy-last-view retains moving positions at 1.87 s.The predictor reads stationary objects in 0–87% of scenes from 0.27 s on; copy-last-view retains stationary objects at 0.80–1.00 throughout.Source: I SYMMETRIC READOUTS, MATCHED CONTROLS… · p. 20
• At 0.53–2.93 s, ViT-H’s moving-ball projection is 0.43–0.33, below the midpoint and 37–60% as far above control as copy-last-view.For a ball inside a closed box, the projection is 0.35 at 0.53 s and 0.16 at 2.93 s; the trace is specific to the ball’s position.Source: I SYMMETRIC READOUTS, MATCHED CONTROLS… · p. 21
• Under V-JEPA’s tube query, released ViT-H loses the ball within 0.8 s in dense context, but retains it for four tubelets when 90% of other context is hidden.ViT-L and ViT-g do not gain carry-over from the sparse-context query; continued tube-mask training makes the query answerable, not evidence of a query artefact in the released model.Source: I SYMMETRIC READOUTS, MATCHED CONTROLS… · p. 21
• Across 7.5–30 fps, moving-object decisions remain short-lived: ViT-L loses the ball within one tubelet at every rate, while ViT-H’s decision lasts to 0.27 s at 15 and 30 fps.ViT-g retains it for one or two steps; stationary-object readouts fluctuate across rates.Source: I SYMMETRIC READOUTS, MATCHED CONTROLS… · pp. 21–22
• With two context frames, IntPhys-2019 accuracy is 84.2% for released ViT-H versus 94.3% for the moving curriculum.The static and non-persistent curricula score 93.8% and 88.7%; under the minimum-over-context rule, released ViT-H scores 93.3% and the moving curriculum 90.6%.Source: I SYMMETRIC READOUTS, MATCHED CONTROLS… · p. 22, Table 10 · p. 23
• On hidden moving balls, the moving curriculum reads 0.88 at 1.07 s and 0.38 at 2.13 s; all three curricula read stationary balls at 0.88–1.00 through 3.47 s.None retains a ball in a closed box; with two context frames, the benchmark ranks static with moving, while the probe ranks static last on moving objects.Source: I SYMMETRIC READOUTS, MATCHED CONTROLS… · p. 22
• Curriculum gains survive both IntPhys-2019 scoring rules, but their rankings change; the minimum-rule implementation does not reproduce published numbers.IntPhys 2 shows no gain: released ViT-H reads 52.8–54.0%, while the container curriculum reads 51.2–54.2%.Source: I SYMMETRIC READOUTS, MATCHED CONTROLS… · pp. 22–23

Scope of the claims

• “Video world models” here means the three tested prediction paradigms, not the field; action-conditioned, diffusion, explicit-memory, and other listed models were not probed.VideoMAE was tested only as VideoMAE v1, and Cosmos only at 256×256, below its training resolution; the authors limit those readouts to narrow statements.Source: J SCOPE, LIMITATIONS AND WHAT WOULD… · pp. 23–25

Where it was probed

• Belief curves use one renderer with a single row and speed; no released-model two-world test on real occlusion video was conducted.Natural-video evidence is limited to curriculum evaluation over natural backgrounds and composited coins on real hands; the released encoder itself loses the coin when the hand closes.Source: J SCOPE, LIMITATIONS AND WHAT WOULD… · p. 23

Readout interpretation

• Decision labels and projection traces are distinct readouts, and the paper lacks a known-posterior calibration to map P(present), e, and α onto belief.Carry-over durations are threshold crossings from 60 scenes, resolved to about one tubelet, with no interval reported.Source: J SCOPE, LIMITATIONS AND WHAT WOULD… · pp. 23–24

Query geometry

• The main mask hides the band only after the event, unlike pretraining masks; longer histories and other speeds remain untested, and query geometry changes ViT-H’s carry-over.The tube query leaves about 90% of the frame visible; hiding 90% of surrounding context yields 1.1 s of carry-over for ViT-H, but not ViT-L or ViT-g.Source: J SCOPE, LIMITATIONS AND WHAT WOULD… · p. 24

Mechanism

• The proposed same-instant filling mechanism is a hypothesis, not a demonstrated cause; the effects of clip length, masking ratio, EMA initialization, and clip count are unmeasured.Different mask and data mixtures yielded different curves, but no isolated factor explains the change.Source: J SCOPE, LIMITATIONS AND WHAT WOULD… · p. 24

Interventions

• The 3,000-step synthetic curricula train and test on the same renderer, so the probe cannot separate learned persistence from domain adaptation or copying.Representation costs were not evaluated; the real-hand recipe’s 27–31% IntPhys-2019 accuracy is evidence that some cost exists.Source: J SCOPE, LIMITATIONS AND WHAT WOULD… · p. 24

Benchmark scope

• IntPhys-2019 comparisons use 360 development pairs without a held-out split; under the minimum rule, reported rule-dependence applies to these checkpoints and implementation.IntPhys 2 shows no gain, and released ViT-L and ViT-g scores are far below ViT-H, with the ViT-g result near chance and unexplained.Source: J SCOPE, LIMITATIONS AND WHAT WOULD… · p. 24

Statistical and evidential boundaries

• Many time-point and condition comparisons receive no correction, so p-values indicate paired separation strength rather than confirmation of preregistered hypotheses.A recipe ablation, held-out renderer, real-occlusion test, and benchmark implementation reproducing published minimum-rule numbers are among the measurements the authors say would change their conclusions.Source: J SCOPE, LIMITATIONS AND WHAT WOULD… · pp. 24–25