Source-linked AI summary
PhysWAM: Physically Consistent World Action Model for Autonomous Driving
Dhruv Parikh, Fengcheng Yu, Quankai Gao, Jiawei Yang, Junjie Ye, Maulik Bhatt, Thang Vu, Charles Ochoa, Rowan McAllister, Igor Vasiljevic, Rajgopal Kannan, Viktor Prasanna, Vitor Guizilini, Yue Wang
TL;DR
Joint prediction of future scenes and ego motion does not by itself constrain them to share geometry. PHYSWAM co-denoises video, metric depth, and ego motion, coupling depth and motion against measured LiDAR geometry; it achieves strong planning and zero-shot transfer while generating accurate depth and coherent video.
Problem
Jointly predicting future observations and ego motion does not necessarily impose a shared geometric constraint on the predictions.
Source: §1 · p. 2Method
PHYSWAM co-denoises multiview video, metric depth, and ego motion, using CPP to compare generated depth transformed by generated motion with LiDAR under recorded motion.
Source: §1 · p. 2Results
PHYSWAM achieves strong planning and zero-shot closed-loop transfer, generates accurate metric depth and temporally coherent video, and CPP improves planning and depth prediction.
Source: §1 · pp. 2–3Takeaways & Limitations
The geometric relationship between scene depth and ego motion provides a direct way to couple world and action generation within a unified model.
Source: §1 · p. 3PHYSWAM is evaluated on recorded data and in simulation, not validated for public-road deployment; its trajectories are not meant to control a vehicle without required safety systems.
Source: Ethics Statement · p. 10
Abstract
from arXiv · showhide
World-action models (WAMs) jointly predict how a scene will evolve and how an agent should act, however joint generation alone does not necessarily impose a shared geometric constraint on these predictions. We present PhysWAM, a unified world-action model for autonomous driving that co-denoises multiview video, metric depth, and ego motion within a single flow-matching transformer. To ground world and action generation in measured scene geometry, we introduce Coupled Point Projection (CPP) that unprojects the generated depth into 3D points, transforms them using the generated $\mathrm{SE}(3)$ ego motion, and minimizes their distance to LiDAR points transformed using the recorded ego motion. This geometric constraint promotes physical consistency with the measured scene by jointly supervising generated depth and motion alongside their standard flow-matching objectives. At inference, trajectory selection relies only on a simple label-free consensus rule, with no learned scorer or simulator feedback. We evaluate PhysWAM across NAVSIM v1 and v2 planning, zero-shot closed-loop transfer, and future video and metric-depth prediction. Despite PhysWAM's simple selection procedure, it achieves strong planning performance and transfers zero-shot to unseen driving environments. It also generates accurate metric depth and temporally coherent video, with CPP improving both planning and depth prediction. Together, these results demonstrate that the geometric relationship between scene depth and ego motion provides a direct way to couple world and action generation within a simple unified model.
1 Introduction
Jointly generating future observations and ego motion does not itself constrain them to share scene geometry. PHYSWAM co-denoises multiview video, metric depth, and ego motion, then couples depth and motion against measured LiDAR geometry.
Unified world-action generation
- PHYSWAM co-denoises multiview video, metric depth, and ego motion in one transformer, conditioned on current images, calibration, driving context, and ego history.It jointly generates future video, metric depth maps, and an ego trajectory through iterative denoising.Source: §1 · p. 2
Geometric coupling
- Coupled Point Projection (CPP) transforms generated depth with generated ego motion and compares it with LiDAR under recorded motion, jointly supervising both predictions.The shared geometric loss complements the individual prediction losses and promotes consistency with measured scene geometry.Source: §1 · p. 2
Simple inference
- PHYSWAM selects a representative trajectory by pairwise distances among samples, without labels, a learned scorer, or simulator feedback.The consensus rule is parameter-free.Source: §1 · p. 2
Evaluation and results
- PHYSWAM achieves strong planning and zero-shot closed-loop transfer, while generating accurate metric depth and temporally coherent video.Planning evaluation includes NAVSIM v1 and v2 and zero-shot HUGSIM transfer without target-domain fine-tuning.Source: §1 · pp. 2–3
- CPP improves planning and depth prediction, while consensus selection improves planning over using a single generated trajectory per scene.Together, these results support coupling generated depth and ego motion through a shared geometric objective.Source: §1 · p. 3
2 Related Work
Prior driving work connects future-scene prediction and planning through several routes, while PHYSWAM combines joint generation with a geometric objective linking generated depth and ego motion.
Generative driving world models
World-action models and planning
- Prior approaches use future prediction as an auxiliary policy objective [70], jointly generate video and action [79] [63], or evaluate and refine trajectories using predicted futures [37].PHYSWAM instead selects among generated samples with parameter-free consensus, without a learned scorer or simulator feedback.Source: §2 · p. 3
Geometry in world and action generation
- Driving models use geometry as conditioning [7], generated output [32] [17] [76], planning representation [67] [40], or a video-feature training target [14].PHYSWAM's CPP adds generated ego motion to the geometric objective, comparing projected depth with LiDAR under recorded motion in one frame.Source: §2 · p. 3
3 Method
PHYSWAM jointly generates multiview video, metric depth, and ego motion, then uses geometric losses to connect the generated scene and trajectory to measured geometry. Its training and inference procedures combine those objectives with trajectory constraints and label-free sample selection.
3 Method
- The method combines joint video, depth, and motion generation with a geometric objective that supervises the depth–motion pair against measured scene structure.The section introduces the generation problem, CPP, supporting trajectory constraints, and training and inference procedures.Source: §3 · p. 3
3.1 Problem Formulation and Preliminaries
- The conditional generation problem uses current multiview images, camera calibration, textual driving context, and ego-motion history to generate future images, metric depth, and relative ego motion.The model is trained with rectified flow matching over visual latents and pose coordinates, keeping conditioning frames and motion history clean.Source: §3.1 · pp. 3–4
3.2 Unified World-Action–Geometry Denoising
- A shared transformer jointly denoises multiview RGB, depth, and motion representations, using calibrated camera rays and mRoPE to encode camera geometry and position.RGB and depth share a frozen VAE; visual and action tokens attend bidirectionally to one another.Source: Figure 2 · p. 4, §3.2 · pp. 4–5
3.3 Coupled Point Projection
- CPP unprojects generated depth into 3D and transforms it with generated motion, then compares corresponding points with LiDAR geometry carried by recorded poses.Errors in depth, motion, or both affect the same robust metric loss, which jointly supervises both predictions alongside their individual flow-matching objectives.Source: §3.3 · pp. 5–6
3.4 Training and Inference
- Obstacle and drivable-area hinge losses constrain waypoint positions through generated motion, independently of generated depth, while gradient balancing scales geometric losses against flow-matching gradients.At inference, the model selects the medoid trajectory from multiple jointly denoised samples using pairwise planar distances, without a learned scorer or simulator feedback.Source: §3.4 · p. 6
4 Experiments
Experiments assess PhysWAM’s planning, ablations, and generated-world quality. Results cover NAVSIM planning, zero-shot HUGSIM driving, and future video and metric-depth prediction.
4 Experiments
- NAVSIM experiments use navtrain windows and NAVSIM-only training; HUGSIM evaluation runs all 436 episodes without target-domain training.One sample per planner call is the default; the medoid selects among eight samples.Source: §4 · p. 7, Table 2 · p. 8
4.1 Planning
- 91.4 PDMS and 90.3 EPDMS with one sample per scene place PhysWAM among the highest reported NAVSIM navtest scores without a learned scorer or RL post-training.The medoid of eight samples adds 0.3 PDMS; the per-scene best reaches 95.3 PDMS.Source: §4.1 · pp. 7–8
- 38.1 EPDMS on navhard exceeds the reported world-action models, while zero-shot HUGSIM scores are RC 48.9 and HD-Score 35.5.On HUGSIM’s medium, hard, and extreme scenarios, the best reported HD-Scores remain above PhysWAM’s.Source: §4.1 · p. 8, Table 2 · p. 8
4.2 Ablations
- Without CPP, scores are 89.6 vs 91.4 PDMS, 36.2 vs 38.1 EPDMS, and 33.4 vs 35.5 HD-Score.Without CPP, front-view AbsRel at +2 s is 0.193 vs 0.175, and pose agreement is 70.9% vs 81.4%. Figure 3 illustrates a van collision without CPP versus stopping short with it.Source: §4.2 · pp. 9–10, Figure 3 · p. 9
- In the LoRA ablation, hinges add 1.5 PDMS, CPP adds 2.1, and both together add 2.5.The ablation supports the benefit of the added objective but does not distinguish coupling from metric-space supervision of each branch; that requires a matched decoupled comparison.Source: §4.2 · p. 9
- Generating all three views instead of only the front view raises PDMS by 1.1 overall, with gains of 2.3–2.7 on turns.The gain is 0.3 on straight roads; without generated depth and CPP on side-view cells, the three views add only 0.4 over front view alone.Source: §4.2 · pp. 9–10
- Removing CPP from update 20,000 onward costs 1.2 HD-Score on HUGSIM; both one-sided hinges are annealed to zero by mid-training.The hinges vanish on plans that keep their margins, whereas CPP penalizes deviation on every LiDAR-supported cell.Source: §4.2 · p. 9
4.3 World-Model Quality
- AbsRel 0.175 at +2 s and 0.232 at +4 s measure front-view future depth without scale alignment; generated video reaches FVD-9 111.3 at 600 clips per side.Across 1,200 clips, FVD-9 is 42.2; video-derived camera yaw deviates from generated motion by a median 0.80° over 4 s, versus a 0.29° recorded-clip floor.Source: Table 3 · p. 8, §4.3 · p. 10
5 Conclusion
PHYSWAM jointly generates multiview video, metric depth, and ego motion, coupling depth and motion through CPP. The conclusion reports strong NAVSIM planning, zero-shot closed-loop transfer, and accurate future depth.
5 Conclusion
- PHYSWAM jointly denoises multiview video, metric depth, and ego motion, couples depth and motion with CPP, and reports strong NAVSIM planning, zero-shot closed-loop transfer, and accurate future depth.CPP is used alongside two geometric hinge losses and output-gradient balancing; the authors identify self-supervised coupling, off-trajectory rollout consistency, and larger backbones and scales as future directions.Source: §5 · p. 10
AI Use Statement
- AI models assisted with writing, method implementation, and running and monitoring experiments under the authors’ supervision.The authors state that they hold responsibility for the paper’s content.Source: AI Use Statement · p. 10
Ethics Statement
- PHYSWAM is evaluated on recorded data and in simulation, is not validated for public-road deployment, and requires safety systems before its trajectories control a vehicle.The work uses public datasets and pretrained models and involves no human subjects or additional data collection or annotation.Source: Ethics Statement · p. 10
Reproducibility Statement
- The authors use public NAVSIM, OpenScene, HUGSIM, and Cosmos 3 Nano resources and state that code, scripts, and checkpoints will be released.Appendices provide architecture and objective, data and label construction, hyperparameters, and evaluation protocols.Source: Reproducibility Statement · p. 10
A Additional Method Details
- The method appendix follows Section 3, covering loss representations, the depth head and CPP gradients, hinge-loss distances, output-gradient balancing, and inference.Its notation defines t as scene frames, j as latent frames, and σ as the noise level; x denotes VAE-grid or pose representations, and z denotes transformer tokens.Source: A Additional Method Details · p. 17
A.1 Architecture and Representations
This appendix describes PhysWAM’s shared transformer representations and training supervision, including geometric depth–motion and waypoint losses, gradient balancing, and recorded sensor data in a shared camera frame, as well as inference-time trajectory selection using planar positions.
A.1 Architecture and Representations
- PhysWAM adapts the Cosmos 3 generation pathway while text and generated tokens interact through shared attention, and RGB and depth share visual projections.Depth and ray embeddings distinguish modality and viewing geometry; the text pathway, video VAE, and depth head remain fixed.Source: §A.1 · pp. 17–18
A.2 Depth Head and Coupled Point Projection
- CPP couples generated depth and ego motion by comparing generated 3D points with LiDAR points placed using recorded poses.A fixed depth head maps generated latents to metric depth; one shared residual sends geometric corrections to both depth and motion. CPP adds 1.0 PDMS over flow matching with hinges and 2.1 over flow matching alone.Source: §A.2 · pp. 18–19
A.3 Geometric Hinge Losses
- Two geometric hinges supervise future planar waypoint positions through generated motion alone, encouraging obstacle clearance and interior drivable-area clearance.Obstacle penalties use signed distance so violating waypoints move away from obstacles; drivable-area penalties move them toward the region’s interior.Source: §A.3 · pp. 19–21
A.4 Output-Gradient Balancing
- Output-gradient balancing recomputes geometric-loss weights from output-gradient norms on each update and separately balances CPP’s depth branch.The norms are measured at velocity outputs before scaling, and the depth gain is measured with γ = 1 before the single parameter-update backward pass.Source: §A.4 · pp. 21–22
A.5 Inference and Trajectory Selection
- At inference, generated representations are jointly denoised, decoded RGB and depth are reconstructed, and the depth head is not used.Current RGB, motion history, and text remain fixed during denoising; metric depth is recovered from decoded depth channels.Source: §A.5 · p. 22
- Trajectory selection scores candidates using planar positions alone and returns the chosen candidate in full, without averaging, tie-breaking, or simulator feedback.The selected sample’s video and depth are also outputs of that same joint sample.Source: §A.5 · p. 23
B Data and Labels
- Training uses supervision derived from recorded logs, with geometric labels expressed in the current front-camera frame C0 so depth, motion, and labels share one frame.The appendix covers training windows, dense metric-depth labels, LiDAR targets for CPP, and obstacle and drivable-area labels for hinge losses.Source: B Data and Labels · p. 23
B.1 Source Data and Training Windows
Training uses NAVSIM navtrain logs to construct overlapping four-second windows with forward-facing camera views, recorded motion, and driving context.
B.1 Source Data and Training Windows
- The NAVSIM navtrain split contains 1,192 logs recorded at 2 Hz, with camera images, LiDAR, ego pose, driving command, and tracked 3D boxes.Each window uses the front, front-left, and front-right cameras, with vehicle-specific camera calibration and projections.Source: §B.1 · p. 23
- Each training window includes the current frame, eight future frames spanning 4 seconds, and the preceding frame as motion history; 103,281 windows are used.Windows are formed at every navtrain scene token, overlap heavily, and use no command-based filtering or resampling.Source: §B.1 · p. 23
- The front view is 832 × 468 and side views are 416 × 234; text context names the city, driving command, and current speed.The text is replaced by an empty string in 10% of training examples to support classifier-free guidance.Source: §B.1 · p. 23
B.2 Dense Metric Depth Labels
The appendix describes how dense metric-depth labels are built and checked against LiDAR, then specifies the LiDAR targets and scene labels used for training.
B.2 Dense Metric Depth Labels
- The final depth labels combine LiDAR-anchored MapAnything predictions with corrected fills, bias correction, and verification against LiDAR.On front-view labels, 65% of pixels come from MapAnything, 16% from corrected fill, 15% are sky, and 4% are smooth extensions; after correction, median log-ratio to LiDAR is within 0.023 in every depth band.Source: §B.2 · pp. 23–24
B.3 LiDAR Targets
- LiDAR returns are pooled into 16×16-pixel latent cells, with geometric-mean depth targets and valid correspondences requiring at least three returns.Cells on moving objects are retained because generated and measured geometry describe the same instant at each anchor.Source: §B.3 · p. 24
B.4 Obstacle and Drivable-Area Labels
- Obstacle labels use annotated boxes and filtered static returns, while drivable-area labels rasterize nuPlan road, intersection, and car-park polygons.The drivable map uses a 128×128 grid at 0.5 m resolution and agrees cell-for-cell with the NAVSIM devkit on 60 held-out windows; incomplete training windows are excluded.Source: §B.4 · pp. 24–25
C.1 Hyperparameters and Metrics
The appendix specifies the training objective and planning metrics, then reports how PHYSWAM performs across the two navhard stages.
C.1 Hyperparameters and Metrics
- Training combines flow matching on RGB, depth, and ego motion with CPP and hinge terms, annealing the hinges to zero while keeping CPP active.PDMS combines no-collision, drivable-area compliance, time-to-collision, comfort, and ego progress; EPDMS adds driving-direction, traffic-light, and lane-keeping compliance and splits comfort.Source: §C.1 · p. 25, Table 6 · p. 25
C.2 navhard Stage Scores and Sub-Scores
- In navhard Stage 2, PHYSWAM has the highest no-collision, drivable-area, and time-to-collision terms and the highest stage score; its Stage 1 results lie within the range of other methods.The combined score multiplies the two stages per scene before averaging, rather than combining the stage means.Source: Table 7 · p. 26, §C.2 · p. 26
C.3 HUGSIM by Difficulty, Metric, and Source Dataset
The HUGSIM evaluation shows strong drivable-area compliance across difficulty levels, alongside difficulty- and dataset-dependent performance differences.
C.3 HUGSIM by Difficulty, Metric, and Source Dataset
- PHYSWAM has the highest drivable-area compliance at every HUGSIM difficulty; it leads every metric except comfort on easy scenarios, while UniAD has higher no-collision and time-to-collision terms on medium through extreme scenarios.HD-Score combines route completion with per-step no-collision, drivable-area, time-to-collision, and comfort terms.Source: §C.3 · p. 26
- Route completion and HD-Score are highest on nuScenes and Waymo and lowest on the hard and extreme KITTI-360 scenarios.Overall values average per-dataset means with equal dataset weight within each difficulty before weighting by episode counts.Source: §C.3 · p. 26
- HUGSIM uses a 0.5-second replanning interval, but at 9.4 GPU-seconds per plan the evaluation does not establish real-time operation.The simulator advances only after the planner returns, and the rendered camera rigs differ from training in lens model, mounting height, and appearance without an adapter.Source: §C.3 · p. 26
C.4 Ablation Details
Ablations show that CPP is associated with better trajectory alignment and depth prediction, while generated depth benefits planning through CPP rather than on its own.
C.4 Ablation Details
- Without generated depth and therefore without CPP, the multi-view model scores 87.5 PDMS, 0.3 above flow plus hinge with depth and 0.7 below the full objective.The ablation setting uses rank-128 adapters with α = 256 on generation-pathway attention projections, trained for 16,000 updates.Source: §C.4 · pp. 26–27
- Without CPP, 70.9% of sampled futures end at +4 s within 10° of recorded heading and 0.3 m of recorded position.At 4,000 updates, plans without CPP drift laterally through less accurate heading rather than taking wrong turns.Source: §C.4 · p. 27
- Without CPP, front-view AbsRel at +2 s is 0.193 versus 0.175 with CPP, and depth error is higher across views and horizons.The pixel-level AbsRel increase is 8–15%, and the increase on CPP-supervised cells is 5–7%; video quality is unchanged within sample noise.Source: §C.4 · pp. 27–28, Table 13 · p. 28
- Removing CPP from updates 20,000 onward leaves open-loop scores within sample noise but costs 1.2 HD-Score and 1.1 RC on HUGSIM.Wrong-turn futures rise from 0.4% to 0.6%.Source: §C.4 · p. 28
C.5 World-Model Quality and Consistency
The evaluation reports future metric-depth, front-view video, and consistency measurements against recorded data and LiDAR floors. Generated depth tracks generated motion more closely than recorded motion, while discrepancies grow in sharp turns and over time.
C.5 World-Model Quality and Consistency
- At +2 s and +4 s, front-view depth reaches AbsRel 0.175 and 0.232, respectively, with δ1.25 of 0.814 and 0.742.On the cells supervised by CPP, AbsRel is 0.144 and 0.194.Source: §C.5 · p. 28
- At 600 clips, front-view FVD-9 is 111.3 versus a 91.5 recorded floor and FID is 15.4 versus 11.0; across 1,200 clips, FVD-9 is 42.2 and FID is 6.8.Reported comparison rows use different clip counts and lengths.Source: §C.5 · p. 29, Table 14 · p. 29
- Over 4 s, generated-video yaw deviates from generated motion by a median 0.80° versus 0.29° for recorded clips; the yaw excess is 0.44° with a 95% bootstrap interval of [0.39, 0.48].Above 45°, where the median commanded turn is 58.7°, yaw excess is 2.31°.Source: §C.5 · p. 29, Table 15 · p. 29
- Generated front depth agrees with generated side depth within 1.5× the 0.5 s LiDAR-sweep disagreement, while frame-to-frame disagreement reaches 2.3–3.3× the LiDAR floor over 4 s.Generated depth agrees about 20% more closely with generated motion than recorded motion.Source: §C.5 · p. 29, Table 15 · p. 29
C.6 Sampler Steps and Cost
Sampler-step changes produce small, non-monotonic PDMS differences while increasing plan cost with more steps. Additional qualitative examples show both comparative model behavior and failures of the full model.
C.6 Sampler Steps and Cost
- PDMS shifts by less than a point across sampler counts: 8 steps score 0.7 above the 30-step protocol, while 4 and 15 steps score 0.6 and 0.4 below it; plan cost rises from 3.5 to 9.4 GPU-seconds.The 4- and 15-step differences are within the 0.2–0.3 sample-to-sample standard deviation; other results use 30 steps.Source: §C.6 · pp. 29–30, Table 16 · p. 30
C.7 Additional Qualitative Examples
- Figure 4 compares the model with CPP against the model trained without CPP, and the multi-view model against the single-view model, on two further scenes each; it also shows two scenes in which the full model fails.Source: §C.7 · p. 30
D Limitations
The limitations concern supervision requirements, the scope of geometric consistency, model-scale coverage, and inference cost. The related-work discussion situates PhysWAM among approaches connecting future prediction to planning and adds explicit ego-motion geometry to joint generation.
D Limitations
- CPP training requires LiDAR calibrated to cameras, accurate ego trajectories, dense metric-depth labels, and obstacle and drivable-area labels, tying every objective term to an instrumented platform.Scaling to LiDAR-free driving video would require a self-supervised coupling that the authors have not developed.Source: D Limitations · p. 30
- CPP supervises agreement where LiDAR supports the recorded trajectory; self-consistency under counterfactual commands or beyond the 4 s window is unsupervised, and generalization there remains open..Source: D Limitations · p. 30
- Results use Cosmos 3 Nano at one model size and one training run per configuration, with no run-to-run variation study or evaluation across backbones, sizes, or data scales.The authors attribute the unstudied dimensions to compute constraints.Source: D Limitations · p. 30
- At 30 sampler steps, one plan costs 9.4 GPU-seconds and the medoid rule multiplies cost by sample count; real-time operation and action-only inference were not established.Closed-loop replanning in simulation time does not establish real-time operation.Source: D Limitations · p. 30
E Extended Related Work
- Prior approaches connect prediction to planning through policy objectives [70], joint video-action generation, or trajectory evaluation [37]; PhysWAM adds metric depth and explicit ego-motion geometry.Its shared metric loss compares generated depth transformed by generated motion with LiDAR transformed by recorded motion.Source: E Extended Related Work · pp. 30, 32, Table 17 · p. 32