Source-linked AI summary
GeoWAM: Visual Geometry World Action Models for Autonomous Driving
Yiren Lu, Xin Ye, Jiaming Liu, Jin Yao, Yi-chung Chen, Liam Merino, Dhruva Dixith Kurra, Min Cai, Tom Lampo, Yu Yin, Danhua Guo, Burhan Yaman
TL;DR
Driving world action models often represent scene dynamics in pixels, where geometry and motion remain entangled with visual appearance. GeoWAM instead forecasts future scene geometry and conditions trajectory prediction on those dynamics. Evaluations of geometry prediction and open- and closed-loop planning demonstrate the effectiveness of this visual geometry approach for autonomous driving.
Problem
Pixel-based world action models represent geometry and motion indirectly, making underlying three-dimensional transformations difficult to recover.
Method
GeoWAM pretrains a visual geometry world model to forecast future scene geometry and uses a geometry-conditioned action head to predict future ego trajectories.
Results
Evaluations across future-geometry prediction and open- and closed-loop planning demonstrate the effectiveness of visual geometry world modeling for autonomous driving.
Takeaways & Limitations
Future-geometry prediction provides a pretraining objective that jointly captures spatial structure and temporal evolution for driving policies.
Abstract
from arXiv · showhide
World action models (WAMs) have recently gained increasing attention as a framework for jointly modeling scene evolution and ego actions in autonomous driving. Most existing WAMs learn scene dynamics in pixel space by combining a video-generation backbone for future-observation prediction with an action head for ego-trajectory prediction. Pixels, however, provide only an indirect representation of these dynamics: they entangle geometry and motion with appearance, texture, and illumination, forcing the model to infer three-dimensional transformations from two-dimensional observations. We argue that geometry, represented by point clouds, offers a more natural state space for driving because it explicitly captures spatial structure and the rigid and non-rigid transformations that govern scene evolution while directly aligning with the space in which driving actions are executed. Building on this insight, we introduce \textbf{GeoWAM}, a visual geometry world action model for autonomous driving. Rather than predicting future images, GeoWAM is pretrained to forecast future scene geometry, yielding representations that jointly encode spatial structure and temporal evolution. A geometry-conditioned action head then leverages these learned geometric dynamics to predict future ego trajectories. Extensive open-loop and closed-loop evaluations show that visual geometry world modeling yields substantially stronger driving policies than image-based alternatives, establishing future-geometry prediction as an effective pretraining objective for autonomous driving.
1 Introduction
GeoWAM argues that geometry is a more direct state representation for driving than pixels because it exposes scene transformations and shares a coordinate space with actions. It forecasts future geometry and uses those dynamics for trajectory planning, with evaluations spanning geometry prediction and open- and closed-loop planning.
- Driving agents must anticipate scene evolution because future changes constrain the actions they should execute.
- VLA policies support high-level understanding and decision making, but their action-centric objectives do not explicitly supervise future scene evolution.
- Video world models encode geometry and motion indirectly through appearance, texture, and illumination, leaving underlying three-dimensional transformations difficult to recover.
- Point-cloud geometry explicitly represents three-dimensional structure, reveals temporal transformations, and shares a coordinate space with ego trajectories.
- GeoWAM forecasts future scene geometry from historical multiview observations, then uses a geometry-conditioned action head to predict future ego trajectories.
- GeoWAM is evaluated on future-geometry prediction and open- and closed-loop planning to assess predicted scene structure and planning utility.
2 Related Work
Related work spans video, occupancy, and world action models, alongside general and driving-specific geometric representations. GeoWAM builds on this landscape by forecasting future three-dimensional scene evolution for action modeling.
- Recent driving world models predominantly formulate predictive modeling as future-video generation from past observations and, when available, actions.
- Occupancy-based models forecast voxelized three-dimensional scene evolution together with ego motion, but require voxelized ground-truth occupancy targets.
- World action models couple future-state prediction with action generation so learned environmental dynamics can inform a driving policy.
- General visual geometry models use feed-forward prediction of dense geometric quantities, establishing dense point maps as a scalable representation for three-dimensional reconstruction.
- GeoWAM first forecasts three-dimensional scene evolution from historical multiview images, then uses predicted geometric dynamics to plan ego motion in two training stages.
- Driving-specific geometry models address metric accuracy, temporal motion, surround-view input, and camera variation through metric point maps and ego-pose prediction.
3 Methodology
GeoWAM models future scene geometry from historical multiview observations, then uses geometry-conditioned ego representations to regress future trajectories. Its two-stage design combines geometry forecasting objectives with action planning while preserving the pretrained geometry capability.
- Visual Geometry World Model: GeoWAM pretrains a visual geometry world model to forecast dense future scene geometry from historical multiview images.The model uses an existing geometry encoder to form multilevel historical tokens and learned queries to decode future geometry.
- Visual Geometry World Model: The forecasting branch predicts future feature representations, dense point maps, and per-pixel confidence maps without reconstructing future image appearance.Future geometry is supervised using feature alignment, point-map regression, confidence-aware regression, and surface-normal consistency.
- GeoWAM Action Branch: Future ego-token decoding attends to historical geometry memory and predicted future geometry tokens to produce ego motion consistent with scene evolution.Causal temporal attention models future steps, while a stop-gradient connection prevents trajectory loss from reshaping the predicted geometry.
- GeoWAM Action Branch: An action head combines historical and predicted ego tokens, refines their temporal context, and regresses a single future trajectory of planar positions and headings.The trajectory head uses a learned query and avoids trajectory anchors, mode classification, and iterative sampling.
- Planning Finetuning: Planning finetuning retains current and future geometry objectives while adding trajectory and auxiliary historical-pose regression losses.This jointly supervises geometry preservation, future trajectory prediction, and relative-pose estimation.
4 Experiments
GeoWAM is evaluated on future-geometry prediction and ego-trajectory planning using nuScenes and NAVSIM, including open- and pseudo-closed-loop settings. It achieves strong geometric forecasting and planning performance while preserving coherent scene structure across driving maneuvers.
- Evaluation setup: Experiments evaluate future geometry on nuScenes and planning on NAVSIM v2 navtest and navhard splits.Navhard uses a two-stage pseudo-closed-loop protocol with original and synthetic reactive scenes.
- Future geometry prediction: GeoWAM achieves the lowest Abs Rel at every evaluated horizon, reducing aggregate mean Abs Rel from 0.274 for Epona+DVGT to 0.257.Epona+DVGT is identified as the strongest baseline for this comparison.
- Future geometry prediction: GeoWAM improves mean δ < 1.25 from 0.655 to 0.754, although Epona+DVGT has higher threshold accuracy at one second.GeoWAM performs substantially better from two to four seconds.
- Open-loop planning: GeoWAM reaches an EPDMS of 90.2 on navtest, improving upon DVGT-2 initialization by 0.6 points and achieving the table’s best overall score.It matches the best DDC and TLC scores while remaining competitive on other safety and progress components.
- Pseudo-closed-loop planning: GeoWAM achieves an EPDMS of 36.6 on navhard, outperforming all baseline methods under the pseudo-closed-loop protocol.The protocol feeds rendered observations from predicted ego poses back into the planner, allowing accumulated deviations to affect later predictions.
- Qualitative results: Qualitative visualizations show coherent predicted scene structure, including trees, poles, and fine-grained road markings, across left turns, straight driving, and right turns.Predicted geometry from all future steps is aggregated, with bounding boxes marking successive ego poses.
5 Conclusion
GeoWAM models scene evolution through future geometric features rather than future images, using multiview observations and feature-level and dense point-map supervision. Its inverse-dynamics-like action stage converts predicted geometric dynamics into ego trajectories through a geometry-conditioned action head.
- Conclusion: GeoWAM forecasts future geometric features from historical multiview observations using feature-level and dense point-map supervision.This replaces future-image generation as the representation-learning objective.
- Conclusion: An inverse-dynamics-like formulation infers future ego tokens from predicted geometric dynamics and maps them to an ego trajectory.A geometry-conditioned action head performs the trajectory prediction stage.