PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoState: Learning View-Decoupled Geometry States for 3D World Modeling

Hongchang Liu, Jian Liu, Yujuan Ding, Liang Peng, Xun Jiang, Fei Ma, Xing Xu, Yi Bin

arXiv:2610.22343v1cs.CVcs.AIcs.LGcs.RO

TL;DR

View-dependent GFM features entangle scene evolution with camera-induced variation, making predictive states harder to model. GeoState separates scene geometry from camera-dependent observation and predicts scene states apart from camera motion. It outperforms compared world models across reported Waymo horizons, supports cross-view querying, and generalizes zero-shot to KITTI and nuScenes.

  • Problem

    View-dependent GFM features change with camera viewpoint even when the physical scene is unchanged, mixing camera variation with scene evolution in predictive states.

    Source: §1 · pp. 1–2

  • Method

    GeoState aggregates multiview GFM features into a view-decoupled scene state, decodes geometry using query-camera rays, and models camera evolution separately from scene-state prediction.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    GeoState outperforms compared world models in depth forecasting across reported Waymo horizons and supports geometrically consistent cross-view querying and zero-shot transfer to KITTI and nuScenes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    GeoState provides a scene representation that can be predicted more effectively over time and queried from different viewpoints.

    Source: §5 · p. 10

Abstract

from arXiv · show

Recent 3D world models increasingly use features from geometry foundation models (GFMs) as predictive world states, benefiting from their rich geometric priors. However, the view dependence of these features entangles scene dynamics with camera-induced variation, making the resulting state evolution harder to predict. We propose GeoState, a view-decoupled geometry representation for predictive 3D world modeling that separates scene-state evolution from camera-dependent observation. Specifically, a scene tokenizer aggregates multi-view GFM features into a view-decoupled scene state, which a geometry decoder queries using camera rays. An autoregressive world model then predicts future scene states, while camera motion is modeled separately. The predicted states can therefore be decoded under either predicted or externally specified cameras, while retaining a single context-derived scale throughout the rollout without per-frame scale alignment. Experiments show that GeoState outperforms prior 3D world models on Waymo, supports geometrically consistent cross-view querying, and generalizes zero-shot to KITTI and nuScenes. Project page: https://hongchang-liu.github.io/GeoState-project-page/

1 INTRODUCTION

GFM features change with viewpoint, entangling physical scene evolution with camera-induced variation in predictive states. GeoState separates scene geometry from observation, predicts scene states apart from camera motion, and outperforms compared world models across reported Waymo horizons while supporting cross-view querying and zero-shot transfer.

1 INTRODUCTION

  • Structured 3D states offer a geometrically grounded alternative to pixel prediction, which can be redundant and difficult to keep geometrically consistent.Pretrained GFM features are attractive for these states because they carry geometric priors that generalize across scenes and tasks.Source: §1 · p. 1
  • View-dependent GFM features change with camera viewpoint even for static scenes, forcing predictive models to represent camera variation alongside physical scene evolution.This added variation makes latent dynamics harder to predict over long horizons.Source: §1 · pp. 1–2
  • A scene tokenizer aggregates multiview GFM features into a shared scene state, and a query-conditioned geometry decoder introduces the viewpoint only during decoding.The same state can be queried from different viewpoints without re-encoding the scene.Source: §1 · p. 2
  • The world model predicts scene evolution in GeoState space while camera evolution is modeled separately, allowing decoding with predicted or user-controlled cameras without per-frame scale alignment.The formulation maintains a consistent scene gauge throughout temporal prediction and geometry decoding.Source: §1 · p. 2
  • GeoState outperforms compared world models in depth forecasting across reported horizons, maintains consistent rollout scale, and supports cross-view querying and zero-shot transfer to KITTI and nuScenes.Waymo is the in-domain evaluation dataset.Source: §1 · p. 2

2 RELATED WORK

Related work explores predictive 3D states, geometry foundation models, and compact scene representations; GeoState uses view-decoupled scene states for temporal world modeling.

2.1 3D WORLD MODELS

  • VGGT-World and GeoWAM predict geometry features or states, while FR3D separates ego-motion but retains view dependence; GeoState decouples the predictive state itself.Other 3D world models forecast point clouds, occupancy, occupancy-flow fields, or Gaussian scenes.Source: §2.1 · pp. 2–3

2.2 GEOMETRY FOUNDATION MODELS

  • VGGT-Ω strengthens pretrained geometry features for dynamic scenes, which GeoState reformulates as a compact, view-decoupled scene representation for temporal modeling.The passage describes VGGT-Ω as building on geometry foundation model advances including DUSt3R, Fast3R, CUT3R, and VGGT.Source: §2.2 · p. 3

2.3 COMPACT 3D REPRESENTATIONS

  • LVSM, Surflo, and 3DRAE compress multiview inputs, but primarily target reconstruction or novel-view synthesis; this work tests compact scene representations as predictive temporal states.These works demonstrate representations that do not retain a separate dense representation for each view.Source: §2.3 · p. 3

3 METHOD

GeoState separates scene-state evolution from camera-dependent observation: multi-view geometry features are compressed into a view-decoupled state, queried with camera rays, and predicted autoregressively. The method trains this representation and world model with geometry reconstruction, single-step initialization, and autoregressive rollout objectives.

3.1 METHOD OVERVIEW

  • GeoState constructs view-decoupled geometry states from multi-view observations, then models their temporal evolution separately from camera evolution.The overview assigns state construction to a scene tokenizer and future-state prediction to the world model.Source: §3.1 · p. 3, Figure 1 · p. 4

3.2 VIEW-DECOUPLED GEOMETRY STATE

  • Multi-view GFM features in a shared scene coordinate system are aggregated into a fixed-length GeoState that is independent of any single observation viewpoint.GeoStates use their own timestamp's outputs, while cross-time observations establish a shared coordinate frame.Source: §3.1 · p. 3, §3.2 · pp. 4–5

3.3 QUERY-CONDITIONED GEOMETRY DECODER

  • A query-conditioned decoder combines the scene state with camera-ray tokens to reconstruct depth, allowing the same state to be queried from different viewpoints without re-encoding.It reuses the frozen GFM dense prediction head and converts normalized output to metric depth using the scene representation's context-derived scale convention.Source: §3.1 · p. 3, §3.3 · p. 5

3.4 SCENE-STATE WORLD MODEL

  • The world model predicts future GeoStates autoregressively from a sliding four-state context, with camera evolution handled separately and future states decoded under specified or predicted cameras.A residual update makes scene persistence the prediction prior; its residual head starts equivalent to latent Copy-Last.Source: §3.1 · p. 4, §3.4 · p. 5

3.5 LEARNING OBJECTIVES

  • The tokenizer and decoder are trained for geometry reconstruction and novel-view querying while the GFM and dense head remain frozen.Future target states use growing observation prefixes sharing a first-view coordinate anchor, with context-derived scale calibration rather than target-frame alignment.Source: §3.5 · p. 6
  • Training first uses one-step 4-to-1 prediction with state and depth supervision, then seven-step autoregressive rollout supervised through decoded geometry.Direct state matching is omitted during rollout so states can be optimized for decoded geometry rather than exact GFM-feature agreement.Source: §3.5 · p. 6

4 EXPERIMENTS

Experiments evaluate GeoState on Waymo forecasting, cross-view consistency, zero-shot transfer, and representation and supervision ablations. GeoState outperforms compared methods on Waymo and in zero-shot evaluations, while its ablations examine reconstruction fidelity, state predictability, and supervision choices.

4.2 LONG-HORIZON DEPTH FORECASTING ON WAYMO

  • GeoState outperforms all compared methods on both depth metrics at every reported Waymo horizon, with a 40.4% AbsRel reduction over VGGT-World at 2.0 s.The advantage grows with forecast horizon; the results extend beyond the 1.4 s rollout horizon used during training and retain a single context-derived scale without per-frame alignment.Source: Table 1 · p. 7, §4.2 · pp. 7–8

4.1 EXPERIMENTAL SETUP

  • Waymo evaluation uses 100 held-out windows, and the baselines follow their original scale-handling procedures while GeoState keeps scale calibrated at the last observed FRONT frame throughout rollout.Training uses approximately 140k windows from 800 segments; evaluation also includes 113 eligible KITTI drives and all 150 nuScenes validation scenes.Source: §4.1 · pp. 6–7

4.4 ZERO-SHOT GENERALIZATION

  • Without target-dataset adaptation, GeoState outperforms the compared methods on both depth metrics at all reported horizons on KITTI and nuScenes.Predicted states also outperform camera-matched Copy-Last variants under ground-truth and predicted cameras.Source: Table 2 · p. 9, §4.4 · p. 9

4.5 ABLATION STUDIES

  • Scene Tokenizer and Geometry Decoder reconstructions have comparable accuracy to direct VGGT-Ω depth predictions using ground-truth query cameras.The comparison uses target-time RGB observations and evaluates depth against ground-truth LiDAR.Source: Table 3 · p. 9, §4.5 · p. 9

4.3 CROSS-QUERY GEOMETRIC CONSISTENCY

  • World-model-predicted states closely match observation-encoded states in symmetric reprojection AbsRel across all tested camera perturbations and forecast horizons.The same predicted state is queried from the ground-truth FRONT camera and perturbed virtual cameras with jointly varied translation and rotation.Source: §4.3 · pp. 8–9, Figure 3 · p. 8

4.5 ABLATION STUDIES

  • GeoState achieves comparable reconstruction quality but improves AbsRel and δ1 at every horizon over a feature-compression baseline, indicating an easier-to-predict representation.Both representations use the same world model, training data, forecasting objectives, optimization, and evaluation protocol; the authors suggest GeoState reduces camera-induced variation in state dynamics.Source: §4.5 · p. 9, Table 4 · p. 10
  • One-step state supervision improves forecasting, whereas extending latent matching throughout autoregressive rollout degrades long-horizon performance.The findings support using state matching to initialize local transitions and decoded-geometry supervision for autoregressive predictions.Source: §4.5 · p. 10, Table 5 · p. 10

5 CONCLUSION

GeoState is a view-decoupled geometry state learned from GFM features for predictive 3D world modeling, designed to separate scene evolution from camera-dependent observation.

5 CONCLUSION

  • GeoState learns a view-decoupled geometry state from GFM features for predictive 3D world modeling.Source: §5 · p. 10
  • Separating scene evolution from camera-dependent observation provides a representation that can be predicted over time and queried from different viewpoints.Source: §5 · p. 10
  • The conclusion identifies embodied settings, where predictive scene states could support perception, planning, and interaction in dynamic environments, as a direction for future work.Source: §5 · p. 10

A.1 ARCHITECTURE DETAILS

This section describes implementation details for the Scene Tokenizer, geometry decoder, and world model, with their principal configurations and parameter counts summarized in Table 6.

A.1 ARCHITECTURE DETAILS

  • Implementation details cover the Scene Tokenizer, geometry decoder, and world model used in the main experiments.Source: §A.1 · p. 15
  • Table 6 summarizes the principal architectural configurations and parameter counts.Source: §A.1 · p. 15

A.1.1 GEOMETRY-CONDITIONED MEMORY AND SCENE TOKENIZER

The Scene Tokenizer prepares geometry, camera, and ray features as patch memory, then uses learned latent queries to produce the scene state.

A.1.1 GEOMETRY-CONDITIONED MEMORY AND SCENE TOKENIZER

  • Selected GFM feature levels are projected from 2,048 to 512 channels, concatenated, and fused by an MLP into 512-channel features.Confidence-weighted pooled 3D points are normalized by scene scale, Fourier-encoded, combined with log(1 + c), and projected into geometry embeddings.Source: §A.1.1 · p. 15
  • Source-camera pose and source-ray embeddings are incorporated alongside feature and point embeddings in the patch memory.Camera embeddings encode rotation and normalized translation; source rays use six-dimensional Plücker coordinates.Source: §A.1.1 · p. 15
  • A fusion MLP predicts a correction to the feature embedding, and their normalized sum forms the patch memory.Source: §A.1.1 · p. 15
  • Eight Perceiver layers update 512 learned queries of width 512, and a final LayerNorm produces S ∈R512×512.Source: §A.1.1 · p. 15

A.1.2 QUERY-CONDITIONED GEOMETRY DECODER

The geometry decoder encodes query-camera rays and attends to the shared scene state to produce geometry for the requested view, with the frozen VGGT-Ω DPT head predicting camera z-depth.

A.1.2 QUERY-CONDITIONED GEOMETRY DECODER

  • For a camera-to-world pose and intrinsic matrix, each pixel defines a unit ray direction.Source: §A.1.2 · p. 15
  • Normalized query rays use [d, ¯o×d] and are patchified into 1,014 tokens at 416 × 624 resolution, then concatenated with 512 scene tokens.Source: §A.1.2 · p. 15
  • Scene tokens undergo two scene-only self-attention blocks before joint attention with query-ray tokens in sixteen Transformer blocks.Query-token representations are retained after blocks 4, 8, 12, and 16 for multi-layer decoder features.Source: §A.1.2 · p. 15
  • The decoder passes four feature maps to the frozen VGGT-Ω DPT head to predict camera z-depth, using only the scene state, query camera, and output dimensions.The decoder and DPT head remain fixed during world-model training, while gradients pass through them with respect to the scene state.Source: §A.1.2 · p. 15

A.1.3 RESIDUAL WORLD MODEL

The residual world model uses four observed scene states plus a learned masked prediction state. Factorized temporal and spatial attention processes these tokens, and a zero-initialized residual head initially reproduces Copy-Last.

A.1.3 RESIDUAL WORLD MODEL

  • Four observed states, each shaped 512 × 512, are projected from 512 to 1,152 dimensions, while a learned mask token creates the fifth prediction state.Learned temporal and slot embeddings each have 576 dimensions and are concatenated into a 1,152-dimensional positional embedding.Source: §A.1.3 · p. 16
  • Each Transformer block applies temporal attention across five time positions per slot, then spatial attention across 512 slots per time position, followed by a 1,152 → 4,608 → 1,152 feed-forward network.The sublayers use pre-normalization, residual connections, and dropout.Source: §A.1.3 · p. 16
  • A zero-initialized, bias-free residual head maps the fifth-position tokens back to 512 channels, initially recovering Copy-Last exactly.Source: §A.1.3 · p. 16

A.2 CAMERA PREDICTOR

The camera predictor forecasts future FRONT-camera poses from observed ego-motion and the current FRONT image, and its trajectory supplies future query cameras for GeoState and Copy-Last.

A.2 CAMERA PREDICTOR

  • Future FRONT-camera poses are predicted from observed ego-motion and the current FRONT image.Source: §A.2 · p. 16
  • The predicted trajectory supplies future query cameras for both GeoState and Copy-Last.Source: §A.2 · p. 16
  • Both model variants use the same predicted camera trajectory as their future query-camera sequence.Source: §A.2 · p. 16

A.2.1 ARCHITECTURE

This section details camera prediction, training, scale handling, and evaluation, and examines context-dependent VGGT-Ω features and GeoState forecasting across control and motion-stratified settings. It notes substantial feature variation despite more stable anchor-calibrated metric geometry, an advantage over external baselines for the FRONT-only control, and larger gains on dynamic than static windows under both GT and predicted query cameras.

A.2.1 ARCHITECTURE

  • The 8.99M-parameter camera predictor combines motion-prior pose extrapolation with image-conditioned visual corrections, using historical metric poses and the current FRONT image.It encodes motion vectors and their differences, initializes future poses with constant-body-twist extrapolation, and predicts bounded visual corrections; the correction projection starts at zero, so the model initially behaves as motion-only. Waymo and KITTI use direct multi-horizon prediction, while nuScenes predicts autoregressively at 0.25 s intervals.Source: §A.2.1 · pp. 16–17

A.2.2 TRAINING

  • Camera prediction training proceeds from motion-backbone optimization to visual-adapter training and then downstream depth-query adaptation, with only the visual adapter updated in the final stage.The final stage decodes a fixed last-observed GeoState under predicted future cameras and supervises depth against future Waymo LiDAR, alongside trajectory loss and a parameter-anchor regularizer.Source: §A.2.2 · p. 17

A.3 COORDINATE AND SCALE CONVENTIONS

  • The observed context sets the metric and normalized scene scales, which remain fixed across the forecasting rollout without future-target scale re-estimation.The same normalization applies to point coordinates, camera translations, and ray origins; geometry from a separate GFM pass is converted into the context’s normalized physical unit, while intermediate feature vectors are not rescaled.Source: §A.3 · p. 18

A.4 TRAINING DETAILS

  • Waymo validation uses 100 held-out fixed windows, while the training supplements define motion partitions, tokenizer view-dropout sampling, and seven-step World Model targets.The held-out windows comprise 79 Dynamic and 21 Static examples; tokenizer pretraining uses 9,496 scenes, and World Model windows contain four context and seven future states at 0.2 s intervals.Source: §A.4.1 · p. 18, §A.4.2 · p. 18, §A.4.3 · p. 19

A.5.2 INPUT AND QUERY CONSTRUCTION

  • Evaluation shares the World Model state trajectory across GT-camera and predicted-camera queries, scores depth against projected LiDAR, and uses context-derived scale for forecasts.Waymo uses synchronized three-camera observations and KITTI and nuScenes use one view per timestamp; baseline protocols retain their stated per-target alignment or fitting procedures.Source: §A.5.2 · p. 20, §A.5.3 · p. 20

A.5.4 COMPRESSED VGGT STATE BASELINE

  • The appendix also specifies a compressed VGGT-state baseline and dense cross-view consistency evaluation using camera perturbations and bidirectional reprojection.The compressed-state autoencoder has 2.11M trainable parameters and is pretrained on the 9,496-scene dataset before Waymo fine-tuning; consistency evaluation averages across 600,000 query pairs per method and complements physical-ground-truth accuracy.Source: §A.5.4 · p. 21, §A.6.2 · p. 21, §A.6.3 · p. 22

A.7 FEATURE ANALYSIS OF VGGT-Ω

  • VGGT-Ω features encode context-conditioned variation as well as geometry, motivating latent supervision for one-step initialization and decoded-geometry supervision during rollout.Exact latent matching during rollout would require reproducing context-specific feature variation, including directions with limited influence on geometry; decoded-depth supervision instead constrains the geometry needed at inference.Source: §A.7 · pp. 22–23
  • Feature differences rise from 1.38% at layer 4 to 33.55% and 31.15% at layers 17 and 23; the four-layer mean is 19.32%.For the same image, raw DPT outputs differ by 12.08% AbsRel, versus 3.82% after anchor-calibrated metric conversion; feature L2 and depth AbsRel are not directly commensurate.Source: §A.7 · p. 22, Table 8 · p. 23

A.8.1 SINGLE-VIEW-CONTROLLED BASELINE COMPARISON

  • At t+2.0 s, FRONT-only GeoState scores AbsRel 0.143 and δ1 0.813, versus FR3D 0.152/0.800, VGGT-World 0.213/0.731, and DINO-Foresight 0.243/0.663.It outperforms all three external baselines on both metrics at every reported horizon while using a single context-derived scale rather than per-target-frame alignment.Source: §A.8.1 · p. 23

A.8.2 MOTION-STRATIFIED FORECASTING RESULTS

  • On Dynamic windows, GeoState reduces AbsRel versus camera-matched Copy-Last by 13.10% with GT and 9.63% with predicted cameras, with δ1 gains of 0.0236 and 0.0229; Static reductions are 4.92% and 2.17%.Results are frame-wise means over ten horizons; Dynamic includes windows with at least one qualifying visible moving track, and Static comprises the remaining windows. The larger Dynamic gains also occur with GT query cameras, independently of future-camera prediction error.Source: §A.8.2 · pp. 23–24, Table 10 · p. 24
Loading 2610.22343v1…