Source-linked AI summary
SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Driving
Jingyu Li, Junjie Wu, Dongnan Hu, Xiangkai Huang, Bin Sun, Zhihui Hao, Xianpeng Lang, Xiatian Zhu, Li Zhang
TL;DR
VLM-based autonomous driving lacks structured spatial-temporal representations for safe trajectory planning. SGDrive introduces a scene-agent-goal hierarchy within a pretrained VLM, and experiments on NAVSIM report state-of-the-art camera-only performance while revealing limitations under extreme turns from single front-view input.
Problem
VLMs lack specialized 3D spatial-temporal driving understanding, including structured representations of geometry, scene context, and motion patterns needed for safe trajectory planning.
Method
SGDrive structures VLM representation learning around scene geometry, safety-critical agents, and short-term goals, using hierarchical world knowledge to guide trajectory generation.
Results
SGDrive achieves state-of-the-art performance among camera-only methods on NAVSIM, including a PDMS score of 87.4 and strongest scores on collision-related NC and TTC metrics.
Takeaways & Limitations
Hierarchical scene-agent-goal cognition provides a driving-specific representation that supports safer and more reliable trajectory generation.
Takeaways & Limitations
With only a single front-view image, SGDrive can deviate under extreme turns because missing viewpoints make long-horizon prediction difficult.
Abstract
from arXiv · showhide
Recent end-to-end autonomous driving approaches have leveraged Vision-Language Models (VLMs) to enhance planning capabilities in complex driving scenarios. However, VLMs are inherently trained as generalist models, lacking specialized understanding of driving-specific reasoning in 3D space and time. When applied to autonomous driving, these models struggle to establish structured spatial-temporal representations that capture geometric relationships, scene context, and motion patterns critical for safe trajectory planning. To address these limitations, we propose SGDrive, a novel framework that explicitly structures the VLM's representation learning around driving-specific knowledge hierarchies. Built upon a pre-trained VLM backbone, SGDrive decomposes driving understanding into a scene-agent-goal hierarchy that mirrors human driving cognition: drivers first perceive the overall environment (scene context), then attend to safety-critical agents and their behaviors, and finally formulate short-term goals before executing actions. This hierarchical decomposition provides the structured spatial-temporal representation that generalist VLMs lack, integrating multi-level information into a compact yet comprehensive format for trajectory planning. Extensive experiments on the NAVSIM benchmark demonstrate that SGDrive achieves state-of-the-art performance among camera-only methods on both PDMS and EPDMS, validating the effectiveness of hierarchical knowledge structuring for adapting generalist VLMs to autonomous driving.
1. Introduction
SGDrive addresses limitations in VLM-based autonomous driving by structuring driving knowledge hierarchically around scene geometry, safety-critical agents, and short-term goals. This design aims to improve spatial-temporal understanding and trajectory generation for complex traffic scenarios.
- VLM-based driving methods struggle with causal reasoning, high-level scene understanding, and physically safe action generation in complex, long-tail scenarios.
- Existing approaches lack spatial and geometric perception, fail to isolate driving-critical information, and do not model future world-state evolution.
- SGDrive introduces structured 〈world〉 tokens that capture geometric, semantic, and high-level objective information for driving.
- The framework learns scene geometry through occupancy supervision and focuses agent perception on road users likely to influence ego-vehicle motion.
- A DiT trajectory generator progressively refines predictions conditioned on extracted world knowledge for coherent trajectory generation.
- SGDrive combines hierarchical world knowledge, block-wise masked attention, and a DiT decoder, achieving state-of-the-art performance among camera-only methods on NAVSIM.
2. Related works
Related work has evolved from unified end-to-end driving systems toward multimodal VLM and VLA frameworks that connect perception, reasoning, and action. SGDrive extends this direction with hierarchical scene-agent-goal world cognition for safe and reliable driving.
- Unified end-to-end systems jointly address scene perception and ego-trajectory generation, increasingly using multimodal inputs and transformer architectures.
- Driving datasets with natural-language annotations have supported VLM research, including systems that apply chain-of-thought reasoning from perception to planning.
- SGDrive organizes driving cognition hierarchically into scene, agent, and goal representations for safe and reliable driving.
- VLA approaches map camera inputs to driving decisions and align vision-language understanding with driving actions.
3. Method
SGDrive represents driving knowledge hierarchically through scene geometry, safety-critical agents, and short-term goals, then uses this representation to forecast future world states and generate trajectories.
- Problem definition and notation: SGDrive transforms camera inputs, driving instructions, and ego state into compact world representations for future-world forecasting.The representation is extracted through specialized world queries and hierarchical world heads.
- Hierarchical world knowledge representation: The hierarchy separates scene geometry, safety-critical agent states, and short-term driving objectives.These components encode layout, relevant agent motion, and the ego-vehicle’s immediate objective.
- Geometric scene layout perception: Occupancy supervision trains geometric scene representations to capture spatial structure rather than high-level semantic distributions.The geometric loss combines cross-entropy over spatial locations with resampled binary cross-entropy for sparse occupancy.
- Safety-critical agents detection: Agent detection focuses representation capacity on safety-critical vehicles, pedestrians, and cyclists selected by trajectory relevance and camera visibility.The detection objective uses matched classification and regression losses for selected agents.
- Short-term driving goal forecasting: Short-term goal forecasting predicts a target ego-pose approximately 4 seconds ahead to encode high-level driving intentions.The goal is supervised with an L1 loss and separates high-level decision-making from low-level trajectory planning.
- Structured block-wise attention mask: Block-wise structured attention prevents cross-category leakage while allowing temporal attention within each knowledge category.The mask separates current and future world subqueries, preserving specialized hierarchical representations.
- Diffusion planner: A diffusion planner conditions waypoint denoising on hierarchical world knowledge and ego state to generate future action trajectories.The planner initializes noisy trajectories from a learned prior based on world queries and historical ego-trajectory information.
4. Experiments
SGDrive is evaluated on NAVSIM through benchmark comparisons, component ablations, attention analysis, and qualitative trajectory studies. Results show strong camera-only performance and gains from hierarchical world knowledge, future-state forecasting, and structured attention.
- Main results: 87.4 PDMS establishes SGDrive as state of the art on the NAVSIM test split among camera-only methods.It surpasses InternVL3-8B and QwenVL2.5-8B by 4.1 PDMS and RecogDrive-8B by 0.6 PDMS.
- Main results: 91.1 PDMS under the same RL configuration surpasses all existing methods, including those using LiDAR inputs.The RL setting also achieves the best NC and DAC among RL-based approaches.
- Main results: The model achieves the best scores on NC and TTC, linking explicit spatiotemporal layout, agent interactions, and short-term goals with collision-related performance.The reported result concerns the key collision-related metrics in the NAVSIM evaluation.
- Ablation study: Adding hierarchical world representation improves PDMS by 2.5 points, while future-world forecasting further raises performance to 85.5 PDMS.Future forecasting also improves NC and EP over the representation-only setting.
- Ablation study: Adding safety-critical agents improves NC and DAC, adding driving goals improves EP, and future-state predictions yield a final 87.4 PDMS.The ablation evaluates scene geometry, agents, goals, and future world-state predictions as planner conditions.
- Attention analysis: Structured attention improves EP and overall PDMS by restricting each world query to same-type information, avoiding cross-category noise and semantic leakage.Default causal attention is described as producing overly conservative behavior and lower driving efficiency.
5. Conclusion
SGDrive structures VLM representation learning around a driving-specific scene-agent-goal hierarchy and uses inferred world knowledge to regulate trajectory generation. Experiments on NAVSIM show effectiveness and state-of-the-art safe-driving performance.
- SGDrive models driving environments through scene geometry, road users, and driving goals while extrapolating their future evolution.
- A structured attention-mask mechanism prevents information leakage and suppresses cross-category noise.
- A DiT-based planner uses inferred driving-world knowledge to regulate trajectory generation.
- Comprehensive NAVSIM experiments demonstrate SGDrive’s effectiveness and state-of-the-art performance in safe driving.
Supplementary Material
Table 6 compares hidden-state fusion methods used in the diffusion planner.
- Table 6 presents a comparison of hidden state fusion methods in the diffusion planner.
6. More experiments
The supplementary experiments compare alternative strategies for fusing hidden states within the diffusion planner and describe the NAVSIM evaluation metrics.
- Exp. (a) incrementally injects hidden states from different subqueries across successive cross-attention layers.
- Exp. (b) assigns distinct cross-attention layers to different subqueries.
- Exp. (c) concatenates all subquery hidden states and enables interaction at every cross-attention layer.
- The Extended PDM Score incorporates Traffic Lights Compliance, Lane Keeping Ability, and Extended Comfort beyond the original PDMS.
- EPDMS combines multiplicative penalties for safety-critical violations with a weighted average of positive performance indicators.
7. Additional visualizations
Additional visualizations show trajectories closely following ground truth in straight-driving and turning scenarios, while single front-view input can produce deviations during extreme turns.
- Predicted trajectories closely follow ground truth in both straight-driving and turning scenarios.
- The visualizations also include several failure cases.
- Single front-view input may cause slight deviations under extreme turning conditions.
- Missing corresponding viewpoints makes long-horizon trajectory prediction challenging and can sometimes lead to lane-change errors.
- Multi-view inputs are identified as a promising direction for mitigating these limitations.