AffordDrive3D: Affordance-Aware World-Action Modeling with Spatial Understanding

1 INTRODUCTION

• Most world-action models predict future appearance, but appearance alone does not explicitly represent the spatial structure needed for driving.Geometry provides relative distance and spatial layout, yet geometric future modeling remains relatively underexplored in VLA-based driving frameworks.Source: §1 · pp. 1–2
• Future geometry alone does not indicate which regions are relevant to ego motion, such as drivable areas and collision-risk regions.The introduction frames action relevance as necessary for aligning future world-state prediction with planning objectives.Source: §1 · p. 2
• 91.3 PDMS and 89.9 EPDMS on NAVSIM are reported as state-of-the-art, with gains of 0.9 PDMS and 1.1 EPDMS over ExploreVLA.The paper also reports that ablations support the effectiveness of temporal affordance forecasting and future geometry modeling for trajectory planning.Source: §1 · pp. 2–3
• AffordDrive3D jointly predicts RGB-conditioned future depth and temporal affordances for drivable and collision-critical regions.It uses future RGB latents as context for depth prediction, and a VLM backbone for affordance prediction and shared learning with trajectory generation.Source: §1 · pp. 2–3
• AffordDrive3D extends future world modeling beyond visual appearance to better align it with the planning objective.The model jointly learns future action-relevant regions and spatial structure.Source: §1 · p. 2

2.1 VLA AND WAM FOR AUTONOMOUS DRIVING

• VLA models connect multimodal scene reasoning with trajectory generation, while world-action models add future prediction to action learning.Prior approaches include video generation, latent dynamics, and video prediction as training-time supervision, but often lack explicit action-relevant spatial constraints.Source: §2.1 · p. 3

2.2 GEOMETRY-AWARE DRIVING POLICIES

• Geometry-aware driving work includes future occupancy, depth, and point-cloud prediction, while AffordDrive3D conditions future depth on RGB latents.Unlike geometry-only modeling, it also learns RGB, depth, affordance, and trajectory objectives; geometry alone does not identify drivable or collision-critical regions.Source: §2.2 · p. 3

2.3 AFFORDANCE-AWARE PLANNING

• Affordance-aware planning work has explored future priors and drivable-area constraints, while AffordDrive3D forecasts current and future drivable and collision-critical regions.This provides temporally structured supervision for how action-relevant space evolves over time.Source: §2.3 · p. 3
• The framework encodes multi-view images, navigation command, and ego state with a shared VLM backbone and learnable tokens.Affordance, future RGB, and RGB-conditioned depth branches shape training, while only the action branch is used at inference.Source: Figure 2 · p. 4

3.1 OVERVIEW

• Given multi-view observations, a navigation command, and ego state, AffordDrive3D jointly predicts the future trajectory, driving affordances, appearance, and geometry through four token groups.All components are optimized jointly through the shared backbone, while inference requires only the action branch to generate the trajectory.Source: §3.1 · p. 4

3.2 RGB-CONDITIONED FUTURE GEOMETRY MODELING

• Future depth prediction uses denoised RGB latents as scene context, letting geometry modeling recover 3D structure associated with predicted visual evolution.Separate decoders handle current geometry from observation-grounded RGB context and future geometry from predicted RGB context; depth latents are supervised against pseudo-ground-truth depth with an L1 loss.Source: §3.2 · p. 5

3.3 TEMPORAL AFFORDANCE FORECASTING

• Horizon- and camera-indexed affordance states produce local drivable-area and collision-critical maps for visual-token regions.Collision-critical targets mark regions containing agents or objects hit by plausible ego trajectories; drivable-area loss weights reliable ground evidence and boundaries, while collision prediction combines class-normalized BCE and Dice loss.Source: §3.3 · pp. 5–6

3.4 ACTION GENERATION AND JOINT LEARNING

• Action-token hidden states condition a DiT trained with flow matching to generate future ego trajectories.Separate lightweight GRU heads also predict trajectories from RGB and geometry tokens under L1 supervision, and the framework is trained end-to-end with weighted affordance, RGB, geometry, action, and auxiliary trajectory losses.Source: §3.4 · p. 6

4.1 EXPERIMENTAL SETUP

• Experiments use NAVSIM’s official PDMS/EPDMS protocols and Qwen3-VL-2B with Wan2.1-Fun-1.3B, followed by RL fine-tuning of the action diffusion model.Source: §4.1 · p. 7

4.2 MAIN RESULTS

• 91.3 PDMS and 89.9 EPDMS are AffordDrive3D’s best reported NAVSIM v1/v2 scores; it beats ExploreVLA by 0.9 PDMS and 1.1 EPDMS.Source: §4.2 · p. 7, Table 1 · p. 8, Table 2 · p. 8
• On a narrow curve, AffordDrive3D stays near the lane center while both baselines go off-road; during lead-vehicle braking, it maintains distance while both collide.Source: §4.2 · pp. 7–8, Figure 3 · p. 9

4.3 ABLATION STUDIES

• The full model reaches 89.6 PDMS versus 88.1 for action-only; affordance, RGB, and geometry alone reach 89.2, 88.9, and 89.1 PDMS, respectively.The full model also has the highest NC, DAC, and EP; the reported ablation results omit RL training.Source: §4.3 · p. 8, Table 3 · p. 9
• Drivable-area AP is 0.983 near-term and 0.971 long-term, while collision-critical AP falls from 0.846 to 0.732 across those horizons.Current-frame AP is 0.986 for drivable areas and 0.918 for collision-critical regions.Source: §4.3 · pp. 8–9, Table 4 · p. 10
• Unfreezing the VLM raises collision-critical AP from 0.782 to 0.912 and drivable-area AP from 0.931 to 0.985, indicating useful affordance information in its original representation and gains from supervision.Source: §4.3 · pp. 9–10, Table 5 · p. 10

• AffordDrive3D jointly learns future appearance, geometry, driving affordances, and action within a shared VLM backbone to align world modeling with planning.Source: §5 · p. 10
• It forecasts drivable and collision-critical regions alongside future geometry, providing explicit spatial structure for trajectory planning.Source: §5 · p. 10
• AffordDrive3D achieves 91.3 PDMS and 89.9 EPDMS on NAVSIM; ablation and qualitative results support the effectiveness of geometry modeling and temporal affordance forecasting for planning.Source: §5 · p. 10

Label grid and timestamps

• Labels cover three camera views on an 18 × 32 merged-token grid at timestamps spaced 0.5 seconds apart from 0 to 4 seconds.Each image is resized to 1024 × 576, and each token corresponds to a 32 × 32 pixel patch; Figure 4 shows example targets.Source: A AFFORDANCE LABEL CONSTRUCTION · p. 14, Figure 4 · p. 15

Drivable area

• Reachable roads include lawful branches within 80 m ahead, while ground evidence is filtered using calibrated rays and predicted depth.Ground behind annotated vehicles remains included so the label describes the road surface beneath them.Source: A AFFORDANCE LABEL CONSTRUCTION · p. 14
• Drivable-area targets encode the fraction of retained ground evidence within the reachable road region, with soft boundary values and reduced weights for sparse ground evidence.Tokens with less than 15% ground evidence are treated as unknown, while frames without a reachable road polygon receive zero weight.Source: A AFFORDANCE LABEL CONSTRUCTION · p. 14

Collision-critical regions

• Collision-critical objects are identified by testing route-compliant trajectories in NAVSIM v1 and retaining objects involved in qualifying at-fault collisions.The trajectory set uses geometric plausibility and Driving Direction Compliance, with a fallback to geometrically plausible trajectories when none satisfy the compliance condition.Source: A AFFORDANCE LABEL CONSTRUCTION · pp. 14–15
• Collision-critical labels are projected object-box overlaps on the token grid, with timestamps lacking visible annotated hazards excluded from training and AP evaluation.An object contributes a positive token when its projected box overlaps the image patch.Source: A AFFORDANCE LABEL CONSTRUCTION · p. 15

Training stages

• After supervised fine-tuning, reinforcement learning updates only the action DiT while freezing the VLM, world model, learnable tokens, and auxiliary heads.SFT runs for 14,140 steps, with the video VAE, CLIP image encoder, and VLM vision encoder frozen.Source: B TRAINING DETAILS · p. 15

RL reward

• NAVSIM v1 uses PDMS directly as the RL reward, while NAVSIM v2 uses a constructed reward because EPDMS Extended Comfort depends on consecutive frames.The v2 reward includes an EC flag evaluated against the deterministic trajectory produced by the current policy for the preceding frame.Source: B TRAINING DETAILS · pp. 15–16

RL optimization

• RL samples 16 trajectories per scene across 128 scenes per step and skips groups whose reward range is below 0.02.The method uses drift-corrected SDE flow matching, online noise-scale adaptation, and a KL penalty to the frozen SFT policy.Source: B TRAINING DETAILS · p. 16

World-model visualizations

• Figure 5 displays predicted future RGB, current and future depth, and drivable-area and collision-critical masks over camera views.Left, front, and right views are spatially concatenated and modeled jointly over time.Source: C MORE VISUALIZATION RESULTS · p. 16

Planning comparisons

• Figure 6 compares BEV trajectories from AffordDrive3D, RGB-based DriveVLA-W0, and an action-only VLA baseline.Eight predicted ego poses are converted into simulated states at 10 Hz using the NAVSIM LQR tracker and kinematic bicycle model.Source: C MORE VISUALIZATION RESULTS · p. 16
• In the BEV comparisons, the ego footprint is shown at the violation state, while surrounding agents use their t = 0 annotations for visual context.Source: C MORE VISUALIZATION RESULTS · p. 16

AP computation

• AP is computed independently for the collision-critical and drivable-area heads at each timestamp by pooling token probabilities across scenes and cameras.The evaluation uses standard non-interpolated AP from ranked probabilities, without a prediction threshold.Source: D AFFORDANCE AP COMPUTATION · p. 16

Evaluation labels

• Collision-critical evaluation excludes unknown scene–timestamp pairs, while drivable-area targets are binarized at Y da > 0.5 and low-ground-evidence tokens remain negative.Source: D AFFORDANCE AP COMPUTATION · p. 16

Additional visualizations

• Figure 5 shows predicted RGB, depth, and affordance masks; Figure 6 compares BEV trajectories against RGB-based and action-only baselines.The visualizations include three camera views and planned trajectories.Source: Figure 5 · p. 17, Figure 6 · p. 17