PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoWM: Efficient Direct World Modeling in Explicit Geometry

Mehrdad Noori, Guile Wu, Sam Hosseini, Dongfeng Bai

arXiv:2610.07381v1cs.CV

TL;DR

World models commonly predict future images or latent representations and then recover geometry, while recursive rollouts make longer-horizon forecasting challenging. GeoWM predicts explicit future geometry directly using a geometry foundation model, a flow-matching transformer, and predicted camera motion. Across four datasets, it outperforms the evaluated world models on geometry forecasting and uses substantially less inference time at longer horizons.

  • Problem

    World models often recover geometry from predicted images or latents and use recursive rollouts, while geometry foundation models reconstruct observed scenes rather than predicting their evolution.

    Source: §I · p. 1

  • Method

    GeoWM turns observed RGB frames into geometric history and conditions a flow-matching transformer on that history, predicted camera motion, and the requested horizon to forecast explicit geometry directly.

    Source: §I · p. 1, §III · p. 2

  • Results

    Across four datasets, GeoWM outperforms evaluated world models on depth, camera pose, and 3D point-cloud forecasting; depth error is 4–58% lower at short and 7–48% lower at long horizons versus the strongest comparable baseline.

    Source: §I · p. 2

  • Takeaways & Limitations

    The results support explicit geometry as a state for world models aimed at tasks that depend on 3D understanding.

    Source: §I · p. 2, §V · p. 8

Abstract

from arXiv · show

Modeling 3D scene geometry and its evolution over time is essential for autonomous driving and robotics. A common paradigm is to use world models to predict future images or latent representations of the environment and subsequently recover geometry from these predictions. However, this paradigm does not explicitly model geometric structure and typically relies on recursive rollouts to reach longer prediction horizons, leading to error accumulation and increasing computational cost. To address these limitations, we present GeoWM, a geometry world model that directly forecasts future scene geometry at specified future horizons without recursive rollout. The key idea is to leverage a geometry foundation model to transform observed RGB frames into a geometric history, which conditions a flow-matching transformer to predict the scene geometry at a specified future horizon. We further show that a lightweight camera-motion predictor can accurately estimate the future viewpoint, and that projecting the observed geometry into the predicted viewpoint provides an effective geometric prior for future geometry forecasting. Extensive experiments on four datasets spanning urban driving, aerial flight, and dynamic manipulation demonstrate that GeoWM outperforms the evaluated world models in forecasting depth, camera pose, and 3D scene geometry, while substantially reducing inference time at longer horizons.

I. INTRODUCTION

Existing world models predict future images or features and recover geometry afterward, while geometry foundation models describe observed scenes rather than their evolution. GEOWM predicts explicit future geometry directly, and outperforms evaluated world models across diverse datasets and horizons.

I. INTRODUCTION

  • Geometry foundation models reconstruct present scenes, while predicting scene evolution as cameras and objects move remains important for planning and control.Image- and latent-prediction approaches subsequently recover geometry, and their geometric accuracy can be challenging for complex 3D scenes.Source: §I · p. 1
  • A geometry foundation model converts observed RGB frames into a geometric history that conditions a flow-matching transformer predicting geometry at a requested future horizon.A lightweight camera-motion predictor estimates the future viewpoint, and projecting observed geometry into it supplies a geometric prior.Source: §I · pp. 1–2
  • GEOWM outperforms evaluated world models across four datasets and three metric families; depth error drops 4–58% at short and 7–48% at long horizons versus the strongest comparable baseline.Its inference time is about 98% lower, and its predicted 3D point clouds are most accurate on all four datasets.Source: §I · p. 2
  • Direct horizon-conditioned prediction avoids recursive rollouts, mitigating error accumulation and keeping forecast cost from growing with the horizon.The method queries each requested future horizon directly.Source: §I · pp. 1–2

II. RELATED WORK

Prior world models generate future video or visual features and recover geometry afterward, while geometry foundation models reconstruct observed scenes. GEOWM instead forecasts explicit geometric quantities directly at requested horizons, using predicted camera motion.

II. RELATED WORK

  • Geometry foundation models such as DUSt3R and VGGT reconstruct depth, point maps, or camera parameters from images, but are not designed to predict geometry changes.GEOWM uses a frozen reconstruction model to convert observed RGB frames into geometry.Source: §II · p. 2
  • Video world models generate future frames and reconstruct geometry afterward; feature and latent models forecast visual representations that are decoded into geometry.Autoregressive feedback in both families retains sequential cost that grows with the horizon, despite training remedies aimed at reducing drift.Source: §II · p. 2
  • Latent geometry models such as VGGT-World forecast frozen-backbone tokens and decode them into geometric outputs, whereas GEOWM forecasts explicit geometric quantities directly without recursive feedback.VGGT-World reaches long horizons through autoregressive rollout; GEOWM also uses predicted camera motion to condition its geometry forecast.Source: §II · p. 2

III. METHOD

GEOWM predicts future depth directly in the output space of a geometry foundation model. Its prediction conditions on observed depth history, a depth map warped into the predicted future viewpoint, and the requested horizon.

III. METHOD

  • A frozen reconstruction backbone extracts depth maps and camera poses from the K most recent RGB frames, and a lightweight predictor extrapolates camera motion to the requested horizon.The last observed depth map is warped into the predicted viewpoint to provide a geometric anchor.Source: §III · p. 2
  • A flow-matching diffusion transformer predicts the depth map at time t+h from depth history, the warped anchor, and horizon h, without recursive feedback.The horizon is an explicit model input; the anchor represents the observation from the predicted future viewpoint.Source: §III · p. 2, Figure 2 · p. 3

A. Problem Formulation

The task predicts future depth and camera pose at a requested horizon from recent observed frames, without feeding shorter-horizon predictions into longer ones. A lightweight camera predictor and forward-projected observed depth provide the conditioning and geometric prior for forecasting.

A. Problem Formulation

  • For a positive horizon h, predict depth at t+h and the camera pose relative to the last observed frame, using the same observed history for every horizon.No shorter-horizon prediction is fed into a longer-horizon forecast.Source: A. Problem Formulation · p. 3
  • Depth is predicted instead of point maps because the camera pixel fixes the viewing ray, leaving depth as the only free per-pixel quantity.The point map follows from predicted depth and pose.Source: A. Problem Formulation · p. 3

B. Camera-Motion Prediction and Geometric Conditioning

  • A lightweight predictor maps the K−1 observed relative motions directly to future poses at every horizon in H.It uses ridge regression for rotation and a small regressor for translation, with at most a few tens of thousands of parameters.Source: B. Camera-Motion Prediction and… · p. 3
  • The observed depth map is forward-projected into the predicted camera using the predicted pose and intrinsics, with nearest-depth visibility handling.The projection is written as W.Source: B. Camera-Motion Prediction and… · p. 3
  • The projected observation provides an estimate of observed static structure in the future camera frame and serves as conditioning for the forecast.It is a geometric prior, not the complete predicted depth map.Source: B. Camera-Motion Prediction and… · p. 3
  • The network predicts the complete future depth map and learns to account for errors in observed depth and predicted motion, scene dynamics, and newly visible regions.The projected observation is used as conditioning only.Source: B. Camera-Motion Prediction and… · p. 3

C. Horizon-Conditioned Flow-Matching Transformer

A diffusion transformer operates directly on depth maps, conditioning on observed depths and a projected future-view anchor. Its horizon-conditioned flow-matching objective predicts clean future depth and weights valid pixels by inverse depth.

Architecture

  • The 206M-parameter transformer processes observed depths, the anchor, and noisy target depth as 16×16 patches, with horizon and flow time injected into every block.It has fourteen pre-norm blocks of width 896 and no learned autoencoder, tokenizer, or pretrained weights beyond the frozen backbone.Source: C. Horizon-Conditioned Flow-Matching… · p. 3

Objective

  • Flow matching interpolates the normalized future-depth target with Gaussian noise, and the network predicts the clean target from the interpolant, flow time, horizon, and conditioning stack.The loss is applied to the velocity implied by the clean-target prediction.Source: C. Horizon-Conditioned Flow-Matching… · pp. 3–4
  • The primary loss uses inverse-depth weights over valid pixels, bounded by a depth floor, to align the objective with relative depth error.This weighting is intended to keep nearby structure from being swamped by the far field.Source: C. Horizon-Conditioned Flow-Matching… · p. 4

D. Training and Inference

Training samples an observed window and a dataset horizon, supervising the same network from observed inputs at each horizon. Evaluation spans four datasets and reports depth, camera-motion, and 3D-geometry metrics; inference avoids recursive rollouts.

D. Training and Inference

  • Each training example pairs an observed window with a uniformly drawn horizon, and the network is supervised at every requested horizon using observed inputs.The transformer is trained from scratch on a single GPU in a few hours.Source: D. Training and Inference · p. 4
  • Inference averages four one-pass predictions from pure noise at τ=0, requiring no ODE integration.The read-out targets the relative-error criterion used by the metrics.Source: D. Training and Inference · p. 4
  • Each horizon needs a fixed number of network evaluations regardless of forecast distance, and different horizons can be batched without computing or feeding back intermediate futures.The predicted camera pose is reported as is; on static-camera DOMINO, the anchor is omitted.Source: D. Training and Inference · p. 4

A. Datasets and Evaluation Protocol

  • The four datasets cover driving, aerial flight, and manipulation, with horizons reaching 4.1 seconds on KITTI and 7.7 seconds on DOMINO.DOMINO tests generalization across seen tasks and robots, novel tasks, a novel robot, and both novel.Source: A. Datasets and Evaluation Protocol · p. 4
  • Evaluation measures target-frame depth AbsRel, aligned camera trajectory ATE/RTE/RRE, and 3D point-cloud Chamfer distance under the stated alignment and sampling protocol.Depth is median-scaled per image except on DOMINO, where raw output is evaluated at true scale.Source: A. Datasets and Evaluation Protocol · p. 4

B. Implementation Details

Implementation uses frozen VGGT-1B to extract observed geometry, with depth scaling and dataset-specific handling. The results table covers future depth, 3D geometry, and camera-trajectory metrics across four datasets.

Implementation Details

  • Frozen VGGT-1B produces observed-frame depth, intrinsics, and poses in one pass; training depth is aligned to its per-window scale, with a training-set constant converting DOMINO depth to metres.Pixels without ground truth are masked, and depth is capped at 80 m on driving datasets and 100 m on TartanAir.Source: B. Implementation Details · p. 4
  • Training uses a 0.4 pure-noise sample fraction and depth floor dmin=0.05; TartanAir omits inverse-depth weighting, and optimization runs for 100k steps with AdamW at 10^-4 learning rate.The setup also uses weight decay 0.01, batch size 16, gradient clipping at 1.0, bf16, and EMA decay 0.999.Source: B. Implementation Details · pp. 4–5
  • Table I reports AbsRel and δ1 for depth, CD for 3D geometry, and ATE/RTE/RRE for camera trajectory, grouped by next, short, mid, long, and all horizons.It compares copy-last, VGGT-World, Cosmos-3, and GEOWM across four datasets.Source: Table I · p. 5

C. Main Results

GEOWM leads the evaluated methods in depth and 3D geometry forecasting across four datasets, and in camera-motion forecasting on the three moving-camera datasets. Its depth-error margin grows at longer horizons, while its camera-motion predictor uses only a few tens of thousands of parameters.

Depth, geometry, and camera-motion forecasting

  • GEOWM has the lowest AbsRel at every horizon group and lowest all-horizon Chamfer distance on all four datasets, plus the lowest ATE and RTE on all three moving-camera datasets.It has the highest δ1 in all but one cell and ties Cosmos-3 on mid-horizon KITTI Chamfer distance and KITTI rotation error.Source: C. Main Results · p. 5
  • On KITTI, long-horizon AbsRel is 17.4 for GEOWM versus 21.2 for Cosmos-3; on TartanAir, it is 40.1 versus 57.5.On DOMINO, GEOWM's all-horizon AbsRel is 9.6 versus 20.5 for Cosmos-3.Source: C. Main Results · p. 5
  • GEOWM's all-horizon Chamfer-distance margin over Cosmos-3 ranges from 3% on KITTI to 39% on DOMINO.GEOWM has the lowest Chamfer distance at every horizon group on all four datasets, with a tie at KITTI mid horizons.Source: C. Main Results · p. 5
  • GEOWM's camera-motion translation ATE is 0.23 versus 0.34 on KITTI, 0.02 versus 0.08 on Cityscapes, and 0.13 versus 0.28 on TartanAir, compared with Cosmos-3.Its predictor has a few tens of thousands of parameters.Source: C. Main Results · pp. 5–6
  • On KITTI, GEOWM matches Cosmos-3 at the first frame (6.6 vs. 6.8) within seed-to-seed variation, while its depth-error margin grows at longer horizons.VGGT-World falls below copy-last from mid horizons on KITTI and up to mid horizons on TartanAir.Source: C. Main Results · p. 5

D. Generalisation to Novel Tasks and Robots

GEOWM retains an advantage on DOMINO when tasks, robots, or both are unseen, with smaller gains as task novelty increases. Its ablations also favor explicit geometry and direct prediction over token reconstruction and recursive rollout.

D. Generalisation to Novel Tasks and Robots

  • GEOWM has the lowest AbsRel and Chamfer distance in every DOMINO generalisation cell; relative to copy-last, it reduces all-horizon error by 54% on seen tasks and robots, 45% on an unseen robot, 37% on unseen tasks, and 31% when both are unseen.First-horizon AbsRel is 4.4–5.1 for GEOWM, while every other method is above 10.Source: D. Generalisation to Novel Tasks and… · p. 6
  • Unseen robots cost less than unseen tasks in relative-to-copy-last error reduction; when both task and robot are unseen, the AbsRel margin narrows to one point.At the longest horizon in the two novel-task blocks, Cosmos-3 has the higher δ1.Source: D. Generalisation to Novel Tasks and… · p. 6

E. Ablation Studies

  • Explicit-geometry forecasting beats both depth-head-token variants by at least 2.8 AbsRel points at short horizons, 1.4 at long horizons, and 1.8 overall, using half as many tokens per frame.The stronger token variant adds a decoded-depth loss and reduces all-horizon error from 17.6 to 16.0, though not at the first frame.Source: E. Ablation Studies · p. 7
  • Removing depth conditioning leaves error worse than copy-last, while the anchor matters most at short horizons and observed depth history matters most at long horizons.Using only the last observed depth map is worse than removing either the anchor or history alone at short horizons and overall.Source: E. Ablation Studies · p. 7
  • Scaling the anchor's predicted translation has its minimum horizon-averaged AbsRel at α=1; removing translation raises error by 10.7% on KITTI and 18.0% on TartanAir.Rotation remains unchanged in this comparison.Source: E. Ablation Studies · p. 7
  • At long KITTI horizons, one-step recursive rollout reaches AbsRel 38.9 versus 17.4 for direct prediction and falls behind copy-last.Predicting two steps from each context halves feedback stages and recovers part, but not all, of the gap; over the full horizon range rollout ends at 2.3 times direct error on KITTI and 1.9 times on TartanAir.Source: E. Ablation Studies · p. 7
  • Accuracy improves from two observed frames to four at every horizon group, with most of the gain already reached at three; other experiments use K = 5.One-step prediction at τ = 0 beats Euler integration at every horizon group, and degradation grows with integration steps; averaging four noise draws gives only a small short- and long-horizon gain and no reported all-horizon change.Source: E. Ablation Studies · p. 7

F. Qualitative Analysis

Qualitative forecasts compare signed relative depth errors and point clouds at short and long horizons on TartanAir and DOMINO.

Qualitative forecasts

  • GEOWM better preserves hospital-room layout as the viewpoint changes and retains raised can and pot geometry, though object boundaries remain smooth.Baseline errors grow around furniture and structural boundaries, and baselines tend to flatten the raised objects on DOMINO.Source: F. Qualitative Analysis · p. 7

G. Computational Cost

GEOWM's inference latency remains flat across evaluated horizons, unlike sequential-rollout baselines, and its conclusion identifies action conditioning and joint motion-geometry prediction as follow-up directions.

G. Computational Cost

  • GEOWM takes 0.11 s across evaluated horizons, while VGGT-World grows from 1.9 s to 36.9 s; reaching the longest horizon is over 300× faster than VGGT-World and over 80× faster than Cosmos-3.Producing all twenty horizons costs 12 TFLOPs; VGGT-World spends 1116 TFLOPs over the same set, while Cosmos-3 takes 4.9 s even at the nearest evaluated horizon.Source: G. Computational Cost · pp. 7–8

V. CONCLUSION

  • GEOWM combines explicit geometric prediction with direct horizon conditioning, and the paper identifies robot-action conditioning and joint motion-geometry prediction as future work.The conclusion reports forecasting across driving, aerial flight, and robotic manipulation, with generalisation to unseen manipulation tasks and robot embodiments.Source: §V · p. 8

Related papers

Loading 2610.07381v1…