SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Driving

• VLM-based driving methods struggle with causal reasoning, high-level scene understanding, and physically safe action generation in complex, long-tail scenarios.Source: §1 · p. 1
• Existing approaches lack spatial and geometric perception, fail to isolate driving-critical information, and do not model future world-state evolution.Source: §1 · p. 2
• SGDrive introduces structured 〈world〉 tokens that capture geometric, semantic, and high-level objective information for driving.Source: §1 · p. 2
• The framework learns scene geometry through occupancy supervision and focuses agent perception on road users likely to influence ego-vehicle motion.Source: §1 · p. 2
• A DiT trajectory generator progressively refines predictions conditioned on extracted world knowledge for coherent trajectory generation.Source: §1 · p. 2
• SGDrive combines hierarchical world knowledge, block-wise masked attention, and a DiT decoder, achieving state-of-the-art performance among camera-only methods on NAVSIM.Source: §1 · p. 2

• Unified end-to-end systems jointly address scene perception and ego-trajectory generation, increasingly using multimodal inputs and transformer architectures.Source: §2.1 · p. 2
• Driving datasets with natural-language annotations have supported VLM research, including systems that apply chain-of-thought reasoning from perception to planning.Source: §2.2 · p. 2
• SGDrive organizes driving cognition hierarchically into scene, agent, and goal representations for safe and reliable driving.Source: Figure 2 · p. 3, §2.2 · p. 3
• VLA approaches map camera inputs to driving decisions and align vision-language understanding with driving actions.Source: §2.2 · p. 3

Problem definition and notation

• SGDrive transforms camera inputs, driving instructions, and ego state into compact world representations for future-world forecasting.The representation is extracted through specialized world queries and hierarchical world heads.Source: §3.1 · pp. 3–4

Hierarchical world knowledge representation

• The hierarchy separates scene geometry, safety-critical agent states, and short-term driving objectives.These components encode layout, relevant agent motion, and the ego-vehicle’s immediate objective.Source: §3.1 · p. 4, §3.3 · p. 4, §3.2 · p. 4

Geometric scene layout perception

• Occupancy supervision trains geometric scene representations to capture spatial structure rather than high-level semantic distributions.The geometric loss combines cross-entropy over spatial locations with resampled binary cross-entropy for sparse occupancy.Source: §3.3 · p. 4

Safety-critical agents detection

• Agent detection focuses representation capacity on safety-critical vehicles, pedestrians, and cyclists selected by trajectory relevance and camera visibility.The detection objective uses matched classification and regression losses for selected agents.Source: §3.3 · pp. 4–5

Short-term driving goal forecasting

• Short-term goal forecasting predicts a target ego-pose approximately 4 seconds ahead to encode high-level driving intentions.The goal is supervised with an L1 loss and separates high-level decision-making from low-level trajectory planning.Source: §3.3 · p. 5

Structured block-wise attention mask

• Block-wise structured attention prevents cross-category leakage while allowing temporal attention within each knowledge category.The mask separates current and future world subqueries, preserving specialized hierarchical representations.Source: §3.3 · p. 5, Figure 3 · p. 5

Diffusion planner

• A diffusion planner conditions waypoint denoising on hierarchical world knowledge and ego state to generate future action trajectories.The planner initializes noisy trajectories from a learned prior based on world queries and historical ego-trajectory information.Source: §3.4 · p. 5

Main results

• 87.4 PDMS establishes SGDrive as state of the art on the NAVSIM test split among camera-only methods.It surpasses InternVL3-8B and QwenVL2.5-8B by 4.1 PDMS and RecogDrive-8B by 0.6 PDMS.Source: §4.2 · p. 7
• 91.1 PDMS under the same RL configuration surpasses all existing methods, including those using LiDAR inputs.The RL setting also achieves the best NC and DAC among RL-based approaches.Source: §4.2 · p. 7
• The model achieves the best scores on NC and TTC, linking explicit spatiotemporal layout, agent interactions, and short-term goals with collision-related performance.The reported result concerns the key collision-related metrics in the NAVSIM evaluation.Source: §4.2 · p. 7

Ablation study

• Adding hierarchical world representation improves PDMS by 2.5 points, while future-world forecasting further raises performance to 85.5 PDMS.Future forecasting also improves NC and EP over the representation-only setting.Source: §4.3 · p. 8
• Adding safety-critical agents improves NC and DAC, adding driving goals improves EP, and future-state predictions yield a final 87.4 PDMS.The ablation evaluates scene geometry, agents, goals, and future world-state predictions as planner conditions.Source: §4.3 · p. 8

Attention analysis

• Structured attention improves EP and overall PDMS by restricting each world query to same-type information, avoiding cross-category noise and semantic leakage.Default causal attention is described as producing overly conservative behavior and lower driving efficiency.Source: §4.3 · pp. 8–9

• SGDrive models driving environments through scene geometry, road users, and driving goals while extrapolating their future evolution.Source: §5 · p. 9
• A structured attention-mask mechanism prevents information leakage and suppresses cross-category noise.Source: §5 · p. 9
• A DiT-based planner uses inferred driving-world knowledge to regulate trajectory generation.Source: §5 · p. 9
• Comprehensive NAVSIM experiments demonstrate SGDrive’s effectiveness and state-of-the-art performance in safe driving.Source: §5 · p. 9

• Table 6 presents a comparison of hidden state fusion methods in the diffusion planner.Source: Table 6 · p. 13

• Exp. (a) incrementally injects hidden states from different subqueries across successive cross-attention layers.Source: §6.1 · p. 13
• Exp. (b) assigns distinct cross-attention layers to different subqueries.Source: §6.1 · p. 13
• Exp. (c) concatenates all subquery hidden states and enables interaction at every cross-attention layer.Source: §6.1 · p. 13
• The Extended PDM Score incorporates Traffic Lights Compliance, Lane Keeping Ability, and Extended Comfort beyond the original PDMS.Source: §6.1 · p. 13
• EPDMS combines multiplicative penalties for safety-critical violations with a weighted average of positive performance indicators.Source: §6.1 · p. 13

• Predicted trajectories closely follow ground truth in both straight-driving and turning scenarios.Source: §7.1 · p. 13
• The visualizations also include several failure cases.Source: §7.1 · p. 13, Figure 9 · p. 16
• Single front-view input may cause slight deviations under extreme turning conditions.Source: §7.2 · p. 13
• Missing corresponding viewpoints makes long-horizon trajectory prediction challenging and can sometimes lead to lane-change errors.Source: §7.2 · p. 13
• Multi-view inputs are identified as a promising direction for mitigating these limitations.Source: §7.2 · p. 13