PaperlayerView exact revision on arXiv

Source-linked AI summary

VGGTWorld-VLA: Intent-Conditioned 3D World Evolution for Autonomous Driving

Zhaoyang Liu, Kun Jiang, Ziying Song, Diange Yang

arXiv:2610.11161v1cs.CVcs.RO

TL;DR

VGGT-World predicts future geometry primarily from observations, leaving driving intentions and actions weakly represented. VGGTWorld-VLA adds semantic and ego-action conditioning through a geometry–language–action bridge. On NAVSIM, it reports improved future-geometry results, while noting semantic-fusion and inference-cost challenges.

  • Problem

    VGGT-World’s future evolution is primarily observation-driven, leaving it unable to adequately distinguish physical futures induced by alternative driving behaviors.

    Source: §1 · pp. 1–2

  • Method

    VGGTWorld-VLA conditions VGGT-World future geometry on VLA-derived driving semantics and ego actions, using a bridge to adapt geometry, semantic, and maneuver and trajectory representations.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    On NAVSIM, it improves latent alignment, depth RMSE, and threshold accuracy over VGGT-World and its NAVSIM-adapted baseline; versus VGGT-World, depth RMSE falls 13.7% and δ1 rises 4.11 percentage points.

    Source: §1 · pp. 2–3, §5 · p. 9

  • Takeaways & Limitations

    The findings support further development of conditional geometric world models for controllable future 3D prediction in autonomous driving.

    Source: §1 · p. 2, §5 · p. 9

    More effective semantic fusion remains an open challenge, and the VLA module’s additional computational cost may limit inference frequency in real-time deployment.

    Source: §5 · p. 9

Abstract

from arXiv · show

VGGT provides a strong foundation for geometry-centric world models by recovering unified 3D scene geometry from visual observations. Although recent extensions enable temporal 3D prediction, their future evolution remains weakly conditioned on driving intentions and actions, limiting their ability to model alternative action-dependent futures. We propose VGGTWorld-VLA, an intention-conditioned extension of VGGT-World for controllable 3D world evolution in autonomous driving. First, we introduce an action--semantic conditioning mechanism that injects complementary driving semantics and ego-motion representations into the future-token stream, enabling different future geometry predictions for the same observed scene under alternative ego actions. Second, we develop a geometry--language--action bridge that adapts historical geometry, VLA semantic features, and maneuver and trajectory representations for joint conditioning of future geometry prediction. We evaluate future geometry prediction on NAVSIM, while conditioning ablations further examine the contributions of semantic and action information. Compared with the baseline, our method demonstrates competitive geometry prediction performance. Ablation studies further support the effectiveness of semantic and action conditioning. These results demonstrate the potential of semantic and action conditioning for controllable VGGT-based world prediction in autonomous driving.

1 INTRODUCTION

VGGT-based work provides geometry-centered representations for driving, but VGGT-World predicts future geometry primarily from observations rather than explicit driving intentions or actions. VGGTWorld-VLA conditions future geometry on driving semantics and ego actions, and reports improved NAVSIM results.

1 INTRODUCTION

  • VGGT unifies several geometry-estimation tasks, and subsequent driving studies extend geometry representations toward reconstruction, dynamic scenes, and planning.The cited examples include DriveVGGT, DVGT, DynamicVGGT, VGGDrive, and DVGT-2.Source: §1 · p. 1
  • VGGT-World predicts temporal geometry from historical observations without explicit driving-intention or action conditioning, limiting its ability to distinguish futures induced by alternative behaviors.The motivation illustrates that an observation-driven forecast may depict straight driving despite an intended left turn.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • VGGTWorld-VLA injects driving semantics and ego-action representations into VGGT-World’s future branch to produce action-dependent future geometry.The framework preserves VGGT-World as its primary world-modeling backbone.Source: §1 · p. 2
  • On NAVSIM, VGGTWorld-VLA reduces depth RMSE by 13.7% and improves δ1 by 4.11 percentage points versus VGGT-World, with comparable AbsRel and a modest increase.The paper also reports improved latent alignment and threshold accuracy over VGGT-World and its NAVSIM-adapted baseline; ablations and action-input interventions examine semantic and action conditioning.Source: §1 · pp. 2–3
  • Its geometry–language–action bridge adapts driving semantics, multi-representation ego actions, and geometry features for joint conditioning of future geometry prediction.This addresses the challenge of combining semantic, action, and geometry representations.Source: §1 · p. 2

2 RELATED WORK

Related work extends VGGT geometry representations across driving reconstruction, world modeling, and vision-language-action policies. These lines of work motivate conditioning future geometry on driving semantics and ego actions.

VGGT for Autonomous Driving

World-Action Models

  • VGGT-World Sun et al. (2026) predicts temporal evolution of frozen VGGT tokens rather than future RGB frames, but remains primarily observation-driven.Other driving world-action models incorporate future images, world tokens, or agent evolution into action learning.Source: §2 · p. 3

Vision-Language-Action Models

  • Vision-language-action models connect semantic reasoning with driving decisions, but most lack an explicit mechanism for predicting how intentions and actions alter future 3D geometry.VGGTWorld-VLA instead uses VLA-derived semantics and ego-action conditions to control VGGT-based world evolution.Source: §2 · p. 3

3 METHOD

VGGTWorld-VLA predicts future geometry tokens from historical geometry, driving semantics, and ego actions. Its bridge adapts these inputs for joint conditioning, while layered action–semantic updates guide future-token prediction and depth supervision.

3 METHOD

  • Historical geometry, ego actions, and driving semantics condition future VGGT geometry tokens, which are decoded into future depth and 3D geometry.Real future images serve only as supervision and evaluation targets; auxiliary trajectory supervision is used during joint fine-tuning.Source: §3 · p. 3, Figure 2 · p. 4

3.1 GEOMETRY–LANGUAGE–ACTION BRIDGE

  • The geometry–language–action bridge preserves geometry, action, and semantic information sources while adapting them for joint conditioning of future geometry.Geometry represents the observed scene, action specifies ego motion, and semantic features supply complementary scene context.Source: §3 · p. 3, §3.1 · p. 4
  • A discrete maneuver descriptor is combined with continuous trajectory features because maneuver categories alone cannot distinguish paths with different turning radii or speeds.Waypoint positions describe the intended path, successive displacements capture motion over fixed intervals, and the normalized index gives temporal position.Source: §3.1 · p. 4
  • Observation-grounded VLA features complement ego-motion conditions with scene semantics and interaction cues, since similar trajectories can occur in different traffic contexts.The semantic branch processes multi-camera observations and a textual prompt; 16 learnable queries aggregate hidden states into fixed-length semantic tokens.Source: §3.1 · p. 5

3.2 ACTION–SEMANTIC 3D WORLD EVOLUTION

  • Historical geometry anchors future-token prediction, while action tokens provide motion conditions and semantic tokens provide complementary driving context.The model predicts future geometry tokens rather than directly generating future RGB images.Source: §3.1 · p. 4, §3.2 · p. 5
  • Eight dual-stream blocks apply separate action and semantic cross-attention updates to future tokens before eight single-stream blocks jointly process historical and conditioned future streams.Action updates precede semantic refinement, but the ordering does not impose strict disentanglement between their effects.Source: §3.2 · p. 5, Figure 3 · p. 6
  • Conditional flow matching connects noise to future geometry tokens by learning a conditional velocity field and predicting endpoint tokens at sampled flow times.The world-model objective combines flow and depth losses; depth supervision uses frozen-teacher future-depth pseudo-targets, with no additional pose or point losses.Source: §3.2 · pp. 5–6

4 EXPERIMENTS

Experiments on NAVSIM assess future geometry prediction against VGGT-derived references and compare the method with VGGT-World baselines. Results show lower RMSE and higher threshold accuracy, while ablations examine action representations, action use, and semantic conditioning.

4.3 MAIN RESULTS

  • Figure 4 contrasts VGGT-estimated reference depth from recorded future frames with predictions conditioned on historical observations, ego actions, and driving semantics.Source: Figure 4 · p. 7
  • RMSE falls 13.7% vs VGGT-World and 15.7% vs VGGT-World-N, while δ1 rises 4.11 and 11.10 percentage points, respectively; AbsRel remains comparable, though VGGT-World-N has lowest AbsRel.The method also achieves the lowest Token MSE and highest cosine similarity; VGGT-World-N improves token-level metrics over the original model but shows inconsistent depth improvements.Source: Table 1 · p. 7, §4.3 · p. 7

4.2 EVALUATION METRICS

  • Future depth is assessed with AbsRel, RMSE, and δ1; latent predictions with Token MSE and cosine similarity; and 3D points with Point EPE and relative Point EPE.Predictions are compared against VGGT-derived references from real future observations.Source: §4.2 · p. 7

4.1 DATASETS

  • NAVSIM evaluation uses 3,000 balanced maneuver clips and pretrained-VGGT predictions from actual future images as pseudo-ground-truth geometry.The subset contains 1,000 clips each for left turns, straight driving, and right turns; all methods use stride 1 and 10 flow-sampling steps.Source: §4.1 · p. 6, §4.3 · p. 7

4.4 ABLATION STUDIES

  • One-hot encoding yields the lowest Token MSE and highest depth threshold accuracy, continuous trajectories lower AbsRel, and adding velocity beats positions alone; Joint gives the strongest overall decoded-geometry performance.Joint combines one-hot maneuvers, positions, and velocity-augmented trajectories, with lowest depth and point errors and competitive threshold accuracy.Source: Table 2 · p. 8, §4.4 · p. 8
  • With history, semantics, and initial noise fixed, correct action inputs achieve the lowest reported errors, while shuffled and wrong-maneuver inputs worsen all metrics, with wrong maneuvers deteriorating most.The interventions change the one-hot maneuver, trajectory positions, and derived velocities on the 3,000-clip motion subset.Source: Table 3 · p. 8, §4.4 · p. 8
  • Figure 5 compares left-turn, straight-driving, and right-turn predictions; scene structures shift rightward under left turns and leftward under right turns.Highlighted structural-boundary and depth-pattern changes provide qualitative evidence that ego actions guide future geometry prediction.Source: §4.4 · pp. 8–9, Figure 5 · p. 9
  • Joint + VLM shows modest gains across reported metrics, extending from latent alignment to decoded depth and 3D points.VLM features provide scene context complementary to action inputs that specify ego motion.Source: Table 4 · p. 9, §4.4 · p. 9

5 CONCLUSION

VGGTWorld-VLA combines driving semantics and ego-motion for future geometry prediction, with NAVSIM results showing improvements on several measures and a modest increase in AbsRel. Semantic fusion and its computational cost remain challenges, while multi-action rollouts and closed-loop evaluation are future directions.

5 CONCLUSION

  • NAVSIM experiments report improved latent alignment, depth RMSE, and threshold accuracy over both VGGT-World baselines, with a modest increase in AbsRel.The action-representation comparison supports combining coarse maneuver templates with continuous motion details; correctly aligned actions improve prediction, and VLM tokens provide modest gains across reported metrics over action-only conditioning.Source: §5 · p. 9
  • Correctly aligned actions improve future geometry prediction, while adding VLM tokens to action-only conditioning yields modest gains across reported metrics.The action-representation comparison supports combining coarse maneuver templates with continuous motion details.Source: §5 · p. 9
  • More effective semantic fusion that preserves geometric prediction quality remains an open challenge, and VLA-module compute may limit real-time inference frequency.Future work includes lightweight conditioning, asynchronous execution, multi-action future rollouts, and closed-loop planning evaluation.Source: §5 · p. 9

AI USE STATEMENT

AI tools assisted with manuscript editing, formatting, and figure layout; the authors attribute the experimental results and scientific contributions to their research.

AI USE STATEMENT

  • AI tools assisted with manuscript editing and LaTeX formatting.The authors state that the experimental results and scientific contributions originate from their research.Source: AI USE STATEMENT · p. 10
  • AI assistance with figure preparation was limited to presentation and layout.The visualizations were derived from actual model outputs.Source: AI USE STATEMENT · p. 10
  • The authors are responsible for verifying the accuracy of the text, references, figures, and reported results.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The work studies future geometry prediction and ego-motion conditioning through offline NAVSIM/OpenScene experiments, without public-road deployment.

ETHICS STATEMENT

  • The study uses the existing NAVSIM/OpenScene driving dataset for offline future-geometry prediction experiments.Source: ETHICS STATEMENT · p. 10
  • The work focuses on scene geometry and ego-motion conditioning rather than identifying individuals.Source: ETHICS STATEMENT · p. 10
  • The experiments do not involve public-road deployment; future autonomous-driving use would require appropriate safety validation.Data use is subject to applicable dataset terms.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The paper describes implementation interfaces, feature construction, training and evaluation details, and data organization to support NAVSIM/OpenScene reimplementation.

REPRODUCIBILITY STATEMENT

  • The reproducibility statement points to model, dataset, baseline, and experimental-setting descriptions, plus appendices covering temporal organization, supervision, optimization, and Euler sampling.Metric definitions and controlled action-input protocols are also documented.Source: REPRODUCIBILITY STATEMENT · p. 10

A IMPLEMENTATION DETAILS

  • The world predictor combines historical geometry, supplied ego motion, and observable driving semantics to predict future VGGT representations.Source: A IMPLEMENTATION DETAILS · p. 13

A.1 REPRESENTATION INTERFACES

  • The geometry branch encodes two historical images from one camera stream as VGGT tokens with 1,024 channels; the flow transformer uses width 512 and eight attention heads.It has eight dual-stream blocks followed by eight single-stream blocks; each dual-stream block applies action then semantic cross-attention to the future stream.Source: §A.1 · p. 13, Table A1 · p. 13

A.2 ACTION FEATURE CONSTRUCTION

  • The Joint action representation combines a one-hot maneuver descriptor with continuous position and motion information.Trajectory features use normalized ego-centric waypoints, successive displacements, and temporal indices, then receive MLP projection and learned embeddings.Source: §A.2 · p. 13

A.3 SEMANTIC FEATURE EXTRACTION

  • The semantic branch processes multi-camera observations and a textual prompt, excludes assistant target responses, and projects normalized language-model hidden states into a 512-dimensional condition space.Sixteen learned queries attend to the sequence while the processor attention mask excludes padding.Source: §A.3 · p. 13

A.4 EVALUATION METRICS

The evaluation compares predicted future geometry with VGGT-derived references across depth, latent-token, and 3D-point spaces. Clips pair historical and future frames from one camera, while independent encoding and a frozen teacher provide historical conditions and future targets.

A.4 EVALUATION METRICS

  • Depth evaluation reports AbsRel, RMSE, and δ1; latent evaluation uses Token MSE and cosine similarity, while point metrics measure 3D correspondence error.Lower AbsRel, RMSE, Token MSE, and point errors indicate better agreement; higher δ1 and cosine similarity indicate better agreement.Source: §A.4 · p. 14
  • Evaluation metrics measure agreement with VGGT-generated references, not accuracy against sensor-measured ground-truth geometry.The references are generated by pretrained VGGT from the complete real image sequence and shared across compared methods.Source: §A.4 · p. 14

B.1 TEMPORAL AND CAMERA ORGANIZATION

  • A four-frame clip selects indices (k −s, k, k + s, k + 2s) from one camera, using the first pair for history and the second for future targets.At the conditioning sample k, semantic inputs and ego actions are associated with the clip; semantic context preserves CAM L0, CAM F0, and CAM R0 while geometry uses one camera.Source: §B.1 · p. 14, Table A2 · p. 14

B.2 HISTORICAL ENCODING AND TEACHER SUPERVISION

  • Independent encoder calls produce historical conditioning and future-frame targets, while a frozen full-sequence teacher supplies future depth and point references.Generated future tokens are concatenated with historical tokens and decoded through the second-stage aggregator and pretrained geometry heads.Source: §B.2 · pp. 14–15

B.3 ENDPOINT PREDICTION AND FLOW TRAINING

Training predicts future-token endpoints under historical, action, and semantic conditions using noisy future tokens and a flow objective. At inference, fixed conditions guide ten descending-noise updates before future tokens are decoded into geometry.

B.3 ENDPOINT PREDICTION AND FLOW TRAINING

  • Training interpolates Gaussian noise and future-token targets, predicts the endpoint under historical, action, and semantic conditions, and minimizes velocity error alongside depth loss.The world objective combines flow loss with depth loss weighted by 0.1; the depth branch decodes the same endpoint at the sampled noise level.Source: §B.3 · p. 15

B.4 EULER INFERENCE

  • Inference keeps conditions fixed while updating future tokens over descending noise levels, then decodes geometry after ten updates.Action comparisons reuse the initial noise tensor, so differences arise from the supplied action condition.Source: §B.4 · p. 15

C.1 TOKEN AND DECODED-GEOMETRY METRICS

The evaluation combines token, decoded-depth, and point measurements with controlled action-input interventions. Supplementary figures show future predictions and action-conditioned depth or token patterns across several urban scenes.

C.1 TOKEN AND DECODED-GEOMETRY METRICS

  • Evaluation uses 3,000 clips spanning left turns, straight driving, and right turns, comparing token and decoded geometry predictions with VGGT-derived references.Depth scores are averaged over two future frames within each clip and then across clips; point errors use the coordinate scale of the VGGT-derived reference.Source: §C.1 · pp. 15–16

C.2 CONTROLLED ACTION INPUTS

  • Action interventions retain, suppress, shuffle, or replace supplied action inputs while holding model parameters, history, semantics, and initial noise fixed.Donor-based conditions transfer action components together, and all conditions are evaluated against the same recorded future reference.Source: §C.2 · p. 16

C.3 SUPPLEMENTARY REPRESENTATION MEASUREMENTS

  • Joint + VLM shows a cosine increase alongside reduced spatially restricted token error compared with Joint in supplementary measurements.Both configurations use the same training budget and evaluation protocol; the near-field measurement follows the reported evaluation mask.Source: §C.3 · p. 16, Table A3 · p. 16

D.1 FUTURE TOKENS AND DECODED DEPTH

  • Figure A1 places last-observed geometry, predicted futures, and VGGT-derived references side by side, with token PCA and depth at both future horizons.The RGB strip shows two historical observations followed by two recorded future images.Source: §D.1 · p. 16, Figure A1 · p. 17

D.2 ACTION-CONDITIONED PREDICTION IN A CURVED-ROAD SCENE

  • The curved-road example compares left, straight, right, and stop predictions from shared history and Euler noise; right-conditioned depth differs visibly from the left-conditioned pattern.The recorded reference is a left turn, and token PCA accompanies depth at the second future frame.Source: §D.2 · pp. 16, 19, Figure A2 · p. 18

D.3 ADDITIONAL URBAN-SCENE EXAMPLES

  • Additional facade and intersection examples show action-conditioned depth patterns across two future horizons against recorded RGB context.The intersection example's right-conditioned second-future prediction follows the prominent central roadside structure visible in the reference.Source: §D.3 · p. 19, Figure A3 · p. 19, Figure A4 · p. 20

Related papers

Loading 2610.11161v1…