PaperlayerView exact revision on arXiv

Source-linked AI summary

PhysiCalWorld: In-Context Physics Calibration for World Models

Yunhao Luo, Wentao He, Yinpei Dai, Hyelin Nam, Hongze Fu, Joyce Chai, Nima Fazeli, Jeong Joon Park

arXiv:2610.30125v1cs.ROcs.AIcs.CV

TL;DR

Existing world models may not infer hidden test-scene physics from frames or short histories, leaving predictions poorly grounded in novel dynamics. PhysiCalWorld uses an exploratory interaction clip encoded as a compact descriptor to calibrate action-conditioned predictions without fine-tuning or analytical physics modeling. Across diverse environments, it improves predictive accuracy and downstream visual planning.

  • Problem

    Frames or short histories may leave hidden physical properties unconstrained, making dynamics prediction ambiguous and prone to drift in test scenes.

    Source: §1 · p. 1, §3.1 · p. 4

  • Method

    PhysiCalWorld compresses a test-scene exploratory interaction sequence into a physics descriptor that conditions subsequent world-model rollouts, trained from video prediction alone without physics labels.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Across diverse environments, PhysiCalWorld substantially improves predictive accuracy and downstream visual planning.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The learned descriptors form a physically structured latent space and extrapolate beyond the physics parameter ranges seen during training.

    Source: §1 · p. 2

    Training requires trajectories grouped by physics configuration, and test-time calibration depends on the quality of the interaction prompt.

    Source: §5 · p. 10

Abstract

from arXiv · show

Existing world models typically perceive a scene through current visual frame or a narrow history window. However, these observations may not reveal the physics properties that determine transition dynamics, such as an object's mass distribution, joint damping, or surface friction. Without calibrating these physics factors for world models, predictions tend to regress toward an empirical prior and induce severe drift in novel test scenes. To this end, we propose PhysiCalWorld, which calibrates action-conditioned world models to the instance-specific dynamics via in-context prompting, without fine-tuning or analytical physics modeling. Our method encodes an exploratory interaction clip collected within the test scene into a compact physics descriptor token, which grounds subsequent predictions in the scene's actual physical behavior. PhysiCalWorld is trained end-to-end with the latent prediction loss alone, without physics labels. At inference time, the descriptor token is extracted once and can be reused across rollouts with negligible compute overhead. We evaluate PhysiCalWorld across four environments covering diverse physics variations and show that it substantially improves rollout prediction accuracy and visual planning success rate. The learned descriptors also form a physically structured latent space and generalize beyond the physics ranges seen during training.

1 INTRODUCTION

Visual frames or short histories may not reveal the physical properties that determine a scene’s dynamics, so world models can make incorrect predictions. PhysiCalWorld uses exploratory interactions to calibrate predictions to test-scene physics without analytical modeling or test-time fine-tuning.

1 INTRODUCTION

  • Current frames or short histories can leave physical properties unconstrained, causing world models to guess among possible futures and predict incorrectly.An opaque bottle can appear identical whether filled or empty yet respond differently to identical actuation.Source: §1 · p. 1
  • PhysiCalWorld conditions a world model on test-scene observation-action pairs compressed into a physics descriptor token.It avoids analytical simulators and test-time gradient descent, and trains end-to-end using video prediction alone without physics labels or a simulator.Source: §1 · p. 2
  • At test time, the descriptor is extracted from a few exploratory interactions and reused across subsequent rollouts without weight updates.The interactions may come from random exploration or scripted policies.Source: §1 · p. 2
  • Across four environments with single and coupled physics variations, PhysiCalWorld improves rollout accuracy and planning success on two world-model backbones.Its single-token descriptor matches or exceeds raw-context conditioning at lower cost; learned descriptors form a physically structured latent space and extrapolate beyond training parameter ranges.Source: §1 · p. 2

2 RELATED WORK

World models are used for robotics tasks including policy evaluation, reinforcement learning, and visual trajectory planning, but narrow observation histories may not identify test-scene dynamics. PhysiCalWorld differs from prior approaches by using a compact interaction-derived descriptor to ground predictions in instance-specific physics.

World Models for Robotics

  • World models support robotics through low-cost policy evaluation, improved reinforcement-learning sample efficiency, policy learning, and visual trajectory planning.Existing methods typically condition on the current frame or narrow observation history, which can leave test-scene dynamics unidentified and lead to drift when physical properties vary.Source: §2 · p. 2

In-Context Learning in Physical AI

  • Prior physical-AI context methods adapt policies using execution history or recovered parameters, while other in-context methods use demonstrations or adapt viewpoints and embodiments.The passage identifies Lee et al., 2020; Kumar et al., 2021; Liu et al., and Yang et al., 2026 among related approaches.Source: §2 · pp. 2–3
  • PhysiCalWorld uses an interaction prompt and compact descriptor for physics-grounded world-model prediction, unlike prior approaches requiring privileged priors or costly raw-context prefixes.The descriptor is designed to minimize computational overhead while maintaining prediction accuracy and rollout efficiency.Source: §2 · p. 3

Physics-Aware Video Generation

  • Physics-aware video-generation methods use equations, simulators, foundation models, or handcrafted rewards, but may assume predefined properties or miss fine-grained instance-level grounding.PhysiCalWorld instead calibrates action-conditioned world models from test-scene interaction without prior assumptions or ground-truth physics supervision.Source: §2 · p. 3

3 METHOD

PhysiCalWorld calibrates action-conditioned world models with a compact descriptor extracted from test-scene interactions, addressing physics that visual observations alone leave underdetermined. It learns the descriptor without physics labels and reuses it to condition predictions at inference.

3 METHOD

  • The method targets action-conditioned prediction in a compact latent space, using encoded visual tokens and action embeddings as inputs to latent dynamics prediction.The described setup supports long-horizon rollout and visual planning.Source: §3 · p. 3

3.1 PHYSICS CALIBRATION VIA PHYSICS DESCRIPTOR TOKEN

  • A compact physics descriptor extracted from exploratory test-scene interactions conditions the world model, addressing dynamics that visual observations alone leave underdetermined.The descriptor is formed from interaction observations and actions, then supplied alongside the ordinary observation-action history.Source: §3.1 · p. 4

3.2 TRAINING PHYSICALWORLD WITHOUT PHYSICS LABELS

  • The context encoder and dynamics predictor are trained end-to-end on latent prediction, using interaction clips and transitions grouped by shared physics configuration rather than physics labels.The method requires no prior physics equations or explicit parameter estimation during training.Source: §3.2 · p. 5

3.3 INFERENCE WITH PHYSICALWORLD

  • At inference, exploratory actions must produce physical contact or motion to reveal scene dynamics; the resulting descriptor can be reused across rollouts and planning candidates while dynamics remain unchanged.Actions may be random or generated by a task-agnostic playful policy, and the descriptor is computed once per scene.Source: §3.3 · p. 5

4 EXPERIMENTS

The experiments test in-context physics calibration across environments with single and coupled physics variations, using unseen configurations for evaluation. PhysiCal improves predictions across these settings, while analysis examines its efficiency, latent structure, and extrapolation beyond training ranges.

4 EXPERIMENTS

  • Evaluation uses 15 unseen configurations per environment, with 75 episodes for open-loop visual metrics and 75 planning tasks per environment.The experiments cover single and coupled physics variations, multiple world-model backbones, and out-of-distribution extrapolation.Source: §4 · p. 6

4.1 ENVIRONMENTS WITH SINGLE PHYSICS VARIATION

  • +PhysiCal substantially outperforms long-history and context-prefix variants in Push-T open-loop rollout, while avoiding their long-context attention cost.It also surpasses +Estimated Physics† across visual metrics without physics labels during training.Source: §4.1 · p. 7
  • In Reacher, in-context conditioning improves both DINO-WM and LeWM, outperforming unprivileged baselines; +PhysiCal beats +Context Prefix on visual metrics and is more compute efficient.Its planning success rate is similar to +Context Prefix, while +Estimated Physics† is the only baseline reported to outperform it overall.Source: §4.1 · pp. 7–8, Table 1 · p. 7
  • In BeerPong, +PhysiCal improves visual prediction over uncalibrated baselines and beats +Context Prefix on visual metrics, but has slightly lower planning success.The authors suggest sparse collision cues may give full-context attention an advantage in identifying collision dynamics.Source: §4.1 · p. 8, Table 2 · p. 8

4.2 ENVIRONMENTS WITH MULTIPLE PHYSICS VARIATIONS

  • In FetchSlide, +PhysiCal yields significant gains across both backbones despite coupled puck mass, surface friction, and air drag, and only +Estimated Physics† surpasses it.The qualitative rollout comparison shows uncalibrated DINO-WM deviating from the target puck position while +PhysiCal captures the ground-truth state.Source: §4.2 · pp. 8–9

4.3 ANALYSIS

  • Against the privileged-physics oracle, +PhysiCal is competitive across most metrics and surpasses it in Push-T, using only an unannotated test-scene interaction clip.The oracle receives ground-truth physics parameters during training and inference.Source: §4.3 · p. 9, Table 3 · p. 9
  • In Reacher, +PhysiCal extrapolates robustly beyond the training damping interval, whereas privileged-physics rollout PSNR drops sharply outside it.The first joint’s damping is swept beyond its training range while the second joint’s damping remains fixed.Source: §4.3 · p. 9, Figure 8 · p. 9
  • In Push-T, descriptors from adjacent unseen centers of mass have strong positive cosine similarity, while distant pairs have negative similarity, despite training without physics labels.The authors interpret this structure as capturing physics properties correlated with scene dynamics.Source: §4.3 · p. 9, Figure 7 · p. 9

5 DISCUSSION

The discussion concludes that in-context calibration improves prediction and visual planning in diverse environments, while identifying data grouping, prompt quality, and stationary physics as limitations.

5 DISCUSSION

  • Training requires trajectories grouped by physics configuration, prompts affect calibration quality, and the framework assumes physics remain stationary within an episode.Changing dynamics, such as fluctuating wind, may require reactive prompting or online descriptor updates.Source: §5 · p. 10
  • Evaluations across diverse environments show substantially improved predictive accuracy and downstream visual planning with physics-grounded prediction for novel test scenes.The approach uses an interaction sequence from the test scene for zero-shot calibration, without finetuning or analytical physics priors.Source: §5 · p. 10

APPENDIX

  • The appendix provides implementation details, environment descriptions, additional quantitative results, speed analyses, and qualitative results.Source: APPENDIX · p. 16

A.1 BASELINES

  • DINO-WM predicts from a short local history without an interaction prompt; its context-prefix variant feeds the full prompt, while long history tests whether 15 frames aid calibration.The context-prefix input increases computation because self-attention scales quadratically with input tokens.Source: §A.1 · p. 16
  • LeWM and TC-LeWM variants test PhysiCal as a plug-in module, while TS-WM evaluates whether temporal regularization promotes physics grounding.TC-LeWM addresses LeWM's weak ball representation in BeerPong; TS-WM uses a curvature regularizer.Source: §A.1 · p. 16
  • Estimated Physics uses supervised physics labels to condition predictions, whereas Privileged Physics supplies ground-truth scene configurations during training and inference.The estimated-physics baseline derives configurations from a context clip; the privileged baseline directly receives true parameters.Source: §A.1 · pp. 16–17

A.2 IMPLEMENTATION OF PHYSICALWORLD

The implementation adds PhysiCal to DINO-WM and LeWM, encoding interaction clips as physics descriptors and conditioning each backbone's predictor.

A.2 IMPLEMENTATION OF PHYSICALWORLD

  • PhysiCalWorld is implemented on DINO-WM and LeWM backbones, with the corresponding base configurations summarized in Table A1.Source: §A.2 · p. 17, Table A1 · p. 17
  • The context encoder maps a 12-frame interaction sequence to a 64-dimensional physics descriptor using frame keypoints, action features, a Transformer, and an MLP.For Push-T, it also embeds the observable pusher state.Source: §A.2 · p. 17
  • Training contexts come from a different episode in the same physics group as the prediction target, and the descriptor conditions each backbone's predictor.Contexts are sampled at each model's prediction gap; DINO-WM broadcasts the descriptor across tokens, while LeWM concatenates it with per-step latents.Source: §A.2 · pp. 17–18
  • Context sampling is environment-specific: Push-T and Reacher use uniform windows, while BeerPong and FetchSlide start at episode onset to capture informative interactions.Short contexts are padded by repeating the final available frame; the encoder's hyperparameters are summarized in Table A2.Source: §A.2 · p. 18, Table A2 · p. 18
  • The TS-WM baseline uses the authors' DINOv2 patch-plus-projection variant with its default curvature regularization and optimization settings.Its hyperparameters are summarized in Table A3.Source: §A.2 · p. 18, Table A3 · p. 18

A.3 TRAINING AND INFERENCE PSEUDOCODE OF PHYSICALWORLD

Training learns a descriptor from interaction patterns using grouped trajectories and prediction loss; inference extracts it once and holds it fixed through autoregressive rollouts.

A.3 TRAINING AND INFERENCE PSEUDOCODE OF PHYSICALWORLD

  • Training uses only physics-configuration group partitions as supervision structure, requiring neither ground-truth physics labels nor an analytical dynamics model.Source: §A.3 · p. 18
  • Because context and target episodes share dynamics within a group, the encoder can learn a descriptor from interaction patterns without knowing the governing properties or their values.The prompt comes from a distinct episode to prevent it from carrying future information about the prediction target.Source: §A.3 · pp. 18–19
  • During training, the predictor receives teacher-forced visual representations with causal masking, and prediction-loss gradients update the predictor and context encoder.The visual decoder is trained separately with an L2 reconstruction loss.Source: §A.3 · p. 19
  • At test time, the descriptor is extracted before rollout and held fixed; autoregressive prediction adds conditioning while preserving base-predictor per-step latency.The algorithm decodes predicted latents to images, while visual planning operates directly in latent space.Source: §A.3 · p. 19

A.4 VISUAL LATENT PLANNING WITH CROSS-ENTROPY METHOD (CEM)

Visual latent planning uses CEM to search action sequences with a physics-calibrated world model, reusing one scene descriptor across optimization iterations.

A.4 VISUAL LATENT PLANNING WITH CROSS-ENTROPY METHOD (CEM)

  • The planner encodes current and goal images, collects an interaction clip, and extracts a scene descriptor that it reuses across CEM iterations.The context can come from random actions or a task-agnostic policy intended to ensure sufficient contacts.Source: §A.4 · p. 20
  • CEM samples action sequences, scores each by the final predicted latent's MSE to the goal latent, and refits its Gaussian distribution to the lowest-cost elite sequences.After repeated optimization iterations, the planner executes the elite-mean action sequence.Source: §A.4 · p. 20
  • FetchSlide and BeerPong use smaller CEM optimization budgets because their action sequences are significantly shorter.Source: Table A4 · p. 20

B ENVIRONMENT TRAINING AND EVALUATION DETAILS

The appendix specifies how physics variation, data collection, and planning are configured across four environments. Additional next-frame results report consistent gains from PhysiCal across both world-model families and all four environments.

B.1 PUSH-T

  • Push-T varies the T-block’s local center of mass while holding appearance fixed, using 135 training and 15 held-out configurations.A CoM-agnostic PPO policy collects trajectories; planning is evaluated with a 25-control-step open-loop budget and with up to two MPC replans.Source: §B.1 · p. 21

B.2 REACHER

  • Reacher varies two joint-damping coefficients across 150 configurations, with random torque actions providing training coverage and 15 configurations held out.Planning success requires both joint errors below 10°; MPC replanning can reduce accumulated model error.Source: §B.2 · p. 22

B.3 BEERPONG

  • BeerPong varies ball-contact damping across 150 configurations, with 135 for training and 15 held out; planning is a one-shot throw evaluated on 75 shared targets.The ball is released with the end effector’s instantaneous velocity, and success requires it to enter and remain in the cup.Source: §B.3 · pp. 22–23

B.4 FETCHSLIDE

  • FetchSlide varies puck mass, surface friction, and fluid density across 150 configurations, with 135 for training and 15 held out.Its one-shot strike task evaluates 75 targets, counting success when the final puck position is within 0.03 m of the target.Source: §B.4 · p. 23

C ADDITIONAL QUANTITATIVE RESULTS

C.1 NEXT-FRAME PREDICTION PERFORMANCE

  • Across four environments, PhysiCal improves next-frame prediction for both DINO-WM and LeWM on PSNR, SSIM, LPIPS, and foreground IoU.Table A6 reports these metrics; the supplied passages do not provide its cell values.Source: §C.1 · p. 24, Table A6 · p. 24

C.2 MPC RESULTS ACROSS EXECUTION BUDGETS

The appendix evaluates MPC by replanning after successive 25-control-step executions, up to 75 steps. PhysiCal improves Push-T results across budgets, while Reacher’s advantage narrows after replanning.

C.2 MPC RESULTS ACROSS EXECUTION BUDGETS

  • In Reacher, PhysiCal has a strong advantage over context-free baselines after 25 steps, but the gap narrows with replanning as MPC compensates for model bias.The authors attribute this to the task’s feedback-correctable structure and two rotary joints.Source: §C.2 · pp. 24–25
  • PhysiCal improves Push-T success across all execution budgets, and Table A7 reports gains for both DINO-WM- and LeWM-based models across allowed replans.The budgets are 25, 50, and 75 control steps, with at most two additional replans.Source: §C.2 · p. 24, Table A7 · p. 25

C.3 MODEL TRAINING EFFICIENCY

The appendix compares training time across world models and reports that PhysiCal adds a modest cost over vanilla DINO-WM on Push-T while remaining faster than the longer-history and context-prefix alternatives.

C.3 MODEL TRAINING EFFICIENCY

  • PhysiCal adds around 11% training cost over vanilla DINO-WM on Push-T, but remains faster than Long History and Context Prefix.For a fair comparison, those alternatives were trained for fewer epochs to approximately align GPU hours; Table A8 notes that training times can vary with cluster conditions.Source: §C.3 · p. 25, Table A8 · p. 25

C.4 MODEL INFERENCE EFFICIENCY

PhysiCal’s compact context encoding keeps inference close to DINO-WM latency in the reported Push-T tests. The appendix also examines success-threshold sensitivity and provides additional qualitative prediction and planning examples.

C.4 MODEL INFERENCE EFFICIENCY

  • PhysiCal takes 16.03 ms for 5-step and 45.83 ms for 20-step open-loop rollouts, close to DINO-WM at 13.78 ms and 44.48 ms, respectively.Its 2.30 ms condition-encoding cost brings next-frame prediction to 7.82 ms, versus 5.58 ms for DINO-WM; context-prefix and long-history rollouts take longer.Source: §C.4 · p. 26, Table A9 · p. 26

C.5 SUCCESS-THRESHOLD SELECTION AND SENSITIVITY

  • Across success thresholds, PhysiCal improves over DINO-WM and LeWM in Reacher and FetchSlide, with larger gains in FetchSlide and modest gains in Reacher.The authors suggest Reacher’s two-joint state space may explain its smaller gains.Source: §C.5 · p. 27, Figure A1 · p. 27

D.1 CONTEXT CLIPS FOR MAIN-TEXT EXAMPLES

  • The appendix includes interaction clips for the main-text Push-T, BeerPong, and FetchSlide examples; context-conditioned models receive the same clip.PhysiCalWorld encodes each clip as a physics descriptor token, whereas Context Prefix feeds the clip directly to the future predictor.Source: §D.1 · p. 28, Figure A2 · p. 28, Figure A3 · p. 28

D.2 UNCROPPED BEERPONG VISUALIZATION

  • The uncropped BeerPong rollout shows DINO-WM + Ours predicting the ball’s bounce accurately, while DINO-WM and DINO-WM + Context Prefix overestimate rebound height.A semi-transparent cyan overlay marks the ground-truth ball position.Source: Figure A5 · p. 29, §D.2 · p. 29

D.3 QUALITATIVE EXTRAPOLATION RESULTS

Across the qualitative extrapolation, open-loop, and planning examples, PhysiCalWorld’s context-conditioned models often preserve or closely match scene motion. The Reacher extrapolation example shows stability through the full rollout while DINO-WM + Privileged Physics deteriorates after t = 7.

D.3 QUALITATIVE EXTRAPOLATION RESULTS

  • DINO-WM + Ours preserves scene structure and predicts arm motion from t = 0 to t = 19, while DINO-WM + Privileged Physics deteriorates after t = 7 in the out-of-range damping example.The first joint’s damping is outside the training range and the second joint’s damping is fixed; predictions are open-loop, without further ground-truth observations.Source: §D.3 · p. 29, Figure A6 · p. 30

D.4 ADDITIONAL QUALITATIVE OPEN-LOOP ROLLOUT RESULTS

  • In Push-T open-loop rollouts, DINO-WM + Ours, LeWM + Ours, DINO-WM + Privileged Physics, and DINO-WM + Context Prefix closely match ground-truth motion; Context Prefix inference is substantially slower.Context Prefix directly attends to the full context sequence.Source: Figure A8 · p. 31
  • In Reacher open-loop rollouts, DINO-WM + Ours, LeWM + Ours, and DINO-WM + Privileged Physics accurately predict arm motion, while Context Prefix overestimates it and remaining baselines underestimate it.The context-conditioned models use the same interaction clip.Source: Figure A10 · p. 32
  • In BeerPong, DINO-WM + Ours, LeWM∗ + Ours, DINO-WM + Privileged Physics, and DINO-WM + Context Prefix reproduce the ball’s bounce; other baselines overestimate rebound, and Context Prefix is substantially slower.LeWM∗ denotes TC-LeWM, which significantly boosts LeWM performance in BeerPong.Source: Figure A12 · p. 33
  • In FetchSlide, DINO-WM + Ours, LeWM + Ours, and DINO-WM + Privileged Physics closely match puck motion; Context Prefix slightly underestimates movement, and baselines without context underestimate it more.The comparison is an eight-frame open-loop rollout.Source: Figure A14 · p. 34

D.5 ADDITIONAL PLANNING RESULTS

  • In DINO-WM-based Push-T planning, DINO-WM + Ours closely matches executed motion, while the remaining models misrepresent T-block translation and rotation.CEM optimizes action sequences toward a goal observation before execution in the environment.Source: Figure A16 · p. 35
  • In LeWM-based Push-T planning, LeWM + Ours accurately predicts executed motion, whereas LeWM underestimates the push at t = 3 and the block moves and rotates farther than predicted at t = 4.The action sequences are generated with CEM to reach a goal observation.Source: Figure A18 · p. 36

Related papers

Loading 2610.30125v1…