STRIKE: Learning Visual State Transitions for Physical World Modeling

• The framework explicitly learns transitions from a current observation, an interaction specification, and elapsed time to predict a future visual state.Source: §1 · pp. 1–2
• A VLM-assisted curation pipeline pairs selected observed training-video states with transition descriptions and elapsed times, grounding supervision in observed changes.The selected images are extracted from training videos rather than synthesized by the annotator.Source: §1 · p. 2
• A timed event planner recursively predicts visual states, whose states and temporal locations condition a separately trained video dynamics model.Source: §1 · p. 2, Figure 1 · p. 2
• STRIKE improves physical-consistency and manipulation-video metrics over corresponding backbone baselines on Physics-IQ Verified, PhyGenBench, Pisa-Experiments, and RoboTwin2.0.Source: §1 · p. 2

2.1 Physics-Aware Generative World Models.

• Physics-aware world models address video generation or control, while STRIKE focuses on explicitly predicting individual interaction outcomes.Source: §2.1 · pp. 2–3

2.2 Keyframe- and Event-Guided Generation.

• KeyWorld [16], SKIP, DCARL [26], RoboEnvision [41], CausalMotion [48], and CoECT [38] use keyframes or events, while STRIKE directly supervises transitions on event-aligned state pairs.Source: §2.2 · p. 3

2.3 Visual Subgoal and State-Transition Prediction.

• MPI [44], SuSIE, and TaKSIE [14] target representations or subgoals, whereas STRIKE supervises interaction-conditioned transitions across physical phenomena.Source: §2.3 · p. 3

3.1 Visual States and Transition Conditions

• The initial visual condition and task context define the rollout input; text-only generation must first synthesize an initial image.Future states occur at ordered timestamps, with τ_k locating each state and Δτ_k measuring the interval from its predecessor.Source: §3.1 · pp. 3–4
• A visual state is an image of a scene configuration, including observable object properties and relations such as contact, support, or attachment.It may depict an interaction boundary or a persistent consequence, but does not explicitly encode quantities such as velocity or occluded contacts.Source: §3.1 · p. 4
• A transition specification describes a local change and its anticipated visible outcome, unlike global context, which describes the overall task or process.The resulting state image shows how that outcome is realized in the particular scene.Source: §3.1 · p. 4

3.2 Conditional Latent Generation

• Conditional image and video generators predict targets in latent representations, while STRIKE retains their generative objectives and changes the prediction targets and conditioning interface.The transition model predicts sparse states under transition and temporal conditions; the video model generates the dense rollout conditioned on those states.Source: §3.2 · pp. 4–5
• The denoising coordinate λ is distinct from video timestamps τ_k and transition intervals Δτ_k.λ parameterizes the prediction target for CogVideoX and Wan, while w_λ is the backbone-specific noise-schedule weighting.Source: §3.2 · pp. 4–5

4.1 Data Annotation and Timed Event Planning

• Training annotation converts videos into time-labeled state transitions, while inference planning predicts local changes and target times from the initial image and task context.Annotation selects informative states and transitions, checks event visibility and caption alignment, and uses extracted video frames; inference has no future reference frames and uses relative intervals for state prediction.Source: §4.1 · p. 5

4.2 Learning and Composing Visual State Transitions

• Qwen-Image-Edit predicts a target state from the source image, transition description, and elapsed time, with temporal RoPE encoding the interval without additional trainable parameters.The source remains uncorrupted and the loss applies only to target outputs; a persistent first-frame sink helps preserve identity, geometry, and appearance during recursive composition.Source: §4.2 · pp. 5–6
• Recursive application of the shared transition model feeds each predicted state into the next step, with the initial image retained as a visual reference.The authors report that adding the first frame as a sink input effectively alleviates exposure bias from accumulated state-generation errors.Source: §4.2 · p. 6

4.3 Video Dynamics Conditioned on Timed Visual States

• A separate CogVideoX or Wan dynamics model generates the full video jointly, conditioned on predicted state images and their temporal locations.State images act as soft conditions, allowing motion between states to depend on the wider predicted sequence rather than independent interval synthesis.Source: §4.3 · pp. 6–7

5.1 Experiment Setup

• Evaluation covers Physics-IQ Verified, PhyGenBench, Pisa-Experiments, and RoboTwin2.0, which assess physical dynamics, commonsense, object trajectories, and manipulation-video metrics.RoboTwin2 val500 contains 500 held-out episodes across 50 tasks and five embodiments.Source: §5.1 · p. 7, §5.2 · p. 9

5.2 Main Results

• 41.9 with CogVideoX-5B and 39.7 with Wan2.2-5B exceed their respective baselines by 11.4 and 20.3 points on Physics-IQ Verified.Both variants outperform baselines on ST-IoU and WS-IoU; CogVideoX reaches 53.5 and 36.0, respectively.Source: §5.2 · p. 7
• 76.91 and 75.62 average PhyGenBench scores improve over the corresponding baselines by 22.12 and 24.79 points for CogVideoX and Wan, respectively.The variants also exceed CausalMotion by 6.70 and 5.41 points, respectively.Source: Table 2 · p. 8, §5.2 · p. 8
• 19.6% lower L2 and 17.3% lower Chamfer Distance than CogVideoX-5B-I2V [43] make the CogVideoX variant best on both Pisa-Experiments distance metrics.It also achieves the highest IoU; both variants outperform evaluated baselines on L2 and Chamfer Distance across 421 evaluation videos.Source: Table 3 · p. 8, §5.2 · p. 8
• STRIKE depicts mug fragmentation after impact and duck displacement after ball contact, while baselines show missed or inconsistent interactions.The comparisons use four time-aligned frames; the paper attributes guidance toward intended physical progression to explicit intermediate states at planned event times.Source: Figure 4 · p. 9, §5.2 · p. 9
• 2.47 and 3.37 EWMScore-15 points plus 43.7% and 42.0% trajectory-score gains over respective backbone baselines are reported on RoboTwin2 val500.Wan reaches 62.53 EWMScore-15 and improves interaction quality and instruction following by 18.2% and 31.8%, respectively.Source: §5.2 · p. 9, Table 4 · p. 10

5.3 Ablation Studies

• Removing the first-frame anchor yields 40.60 Physics-IQ and 0.1528 Pisa-IoU; removing time conditioning yields 41.63 and 0.1617, while excluding synthetic data yields 37.91 and 0.1523.The authors favor all three choices but note that differences in training mixtures and budgets limit strict causal attribution to individual components.Source: Table 5 · p. 10, §5.3 · p. 10
• 24.88 PSNR and 0.863 SSIM on validation states exceed Qwen Image Edit's 17.90 and 0.674, respectively, excluding the initial input image.The learned model also lowers LPIPS and L1 by 57.6% and 63.5%; downstream scores rise from 32.60 to 41.93 on Physics-IQ, 72.50 to 76.91 on PhyGenBench, and 0.1224 to 0.1753 on Pisa-IoU.Source: Table 6 · p. 10, §5.3 · p. 10

6 Conclusion and Limitation

• STRIKE explicitly learns visual state transitions and uses their timed predictions to condition dense video generation.VLM-assisted annotation constructs event-aligned supervision, and a pretrained event planner supplies transition specifications for open-domain inference.Source: §6 · p. 10
• STRIKE is a learned visual predictor rather than an explicit physics simulator, so plausible states do not guarantee correct trajectories, contact dynamics, or conservation laws.Training uses observed states, whereas inference conditions on recursively predicted states, so errors may accumulate over long horizons.Source: §6 · p. 11
• The pipeline adds planning and image-generation costs and depends on the quality of model-assisted annotations.Source: §6 · p. 11

A.1 Datasets

• WISA, PhysicalAI, and PhyCo/Kubric together contribute 130,280 video episodes and 353,840 event-aligned transition pairs.WISA covers 17 annotated categories; PhysicalAI and PhyCo/Kubric add simulated interaction scenarios including contact, momentum transfer, impact, rebound, and deformation.Source: §A.1 · p. 14
• PICA-100K contributes 105,085 instruction-paired source–target images as auxiliary editing supervision, without temporal annotations.The pairs are used with the time condition omitted rather than counted as additional video episodes.Source: §A.1 · p. 14
• OpenVid-1M and WISA sequences provide conditional-dynamics training videos, with prepared CogVideoX and Wan2.2 corpora each containing 69,344 episodes.The clips use 49 frames at 720×480 for CogVideoX and 81 frames at 832×480 for Wan2.2; each backbone reserves 2,000 episodes for internal validation.Source: §A.1 · p. 14
• RoboTwin2.0 provides 124,650 simulated episodes across 50 tasks and five embodiments, with task–seed groups kept within a single split.The 500-episode evaluation covers all tasks and embodiments and measures generalization to unseen episodes of known tasks, not held-out task categories.Source: §A.1 · pp. 14–15

A.2 Event Planning and Data Annotation Pipeline

• Event planning identifies objects and visible processes, decomposes them into minimal subevents, and builds a frame-grounded state plan.Clips without useful visible state or viewpoint progression are flagged as unsuitable for event-based annotation.Source: §A.2 · p. 15, Table 8 · p. 16
• The annotator selects a compact chronological set of event-defining frames, avoiding near-duplicates and annotating each state and intervening transition.Selection prioritizes the earliest frame that clearly establishes a meaningful state or relation.Source: §A.2 · p. 15
• A separate VQA judge checks state claims against visual evidence and aggregates its verdicts to support video-level annotation-quality filtering.Subsequent states are assessed alongside the immediately preceding state as a reference for changes or motion.Source: §A.2 · pp. 15–16

A.3 Training Configurations

• The transition model is fine-tuned on paired source and target states with transition text, using relative timing to condition target-state prediction.It uses frozen image and vision-language encoders, temporal rotary embeddings, and a flow-matching MSE objective on target tokens; inference recursively conditions on generated states.Source: §A.3 · p. 16

A.4 Training and Inference Protocol

• The conditional dynamics models are trained separately on videos paired with observed state images and their temporal indices.At inference, generated transition states replace the observed conditioning states, while the dense video remains the prediction target during training.Source: §A.3 · p. 16, §A.4 · p. 17
• Inference proceeds through event planning, successive state generation, and dense video sampling conditioned on timed memories.The original task context is retained by the video generator, and the external benchmark evaluator supplies no gradients to either learned component.Source: §A.4 · p. 17

A.5 Robotic Manipulation Evaluation

• EWMScore-15 retains 15 metrics across visual and motion quality, content consistency, physics adherence, 3D accuracy, and controllability, while excluding Action Following under the single-instruction protocol.Dataset-level metric scores are normalized to [0, 1], with higher values indicating better performance.Source: §A.5 · p. 17
• All retained metrics receive equal weight, and JEPA similarity is computed over complete generated and reference video collections rather than episode averages.Source: §A.5 · p. 17
• Table 10 provides the prompt for selecting an event-relevant transition-state frame from candidate frames.Source: Table 10 · p. 18

B Additional Results on Benchmarks

• STRIKE pairs a 20B transition model with a lightweight 2B video backbone and remains competitive with, or often surpasses, baselines using 11–16B video models.The 2B CogVideoX version lacks an image-to-video branch, so results use text-to-video.Source: B Additional Results on Benchmarks · pp. 18–19
• Table 11 describes Qwen3-VL verification of whether an extracted transition frame visibly matches its annotated physical state.Source: Table 11 · p. 19
• 65.68 average score is second-best and 66.04 Material score is best on PhyGenBench; on VideoPhy, physical common-sense precision is best at 32.9, Fluid–Fluid precision at 69.1, and semantic accuracy is 70.4.The PhyGenBench results are reported in the T2V setting.Source: B Additional Results on Benchmarks · pp. 18–19, Table 12 · p. 19
• Figure 7 presents qualitative comparisons on PhyGenBench with CogVideoX-5B, Open-Sora-v2, CausalMotion, Cosmos3-Nano, and STRIKE.Source: Figure 7 · p. 20