Source-linked AI summary
SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representation
Wenchao Sun, Xuewu Lin, Yining Shi, Chuang Zhang, Haoran Wu, Sifa Zheng
TL;DR
Existing autonomous-driving systems lose information across modular tasks, while end-to-end methods remain limited by expensive BEV features and simple prediction-planning designs. SparseDrive uses sparse perception and parallel multi-modal prediction and planning, with collision-aware trajectory selection. It reports stronger performance across tasks and higher training and inference efficiency than prior methods.
Problem
Existing modular systems suffer information loss and error accumulation, while end-to-end methods have unsatisfactory performance and efficiency linked to expensive BEV features and straightforward prediction-planning designs.
Method
SparseDrive combines symmetric sparse perception for detection, tracking, and mapping with a parallel motion planner that predicts multi-modal trajectories and selects a safe trajectory hierarchically.
Results
SparseDrive surpasses previous state-of-the-art methods across all tasks and metrics while achieving higher training and inference efficiency.
Takeaways & Limitations
SparseDrive supports sparse scene representation and parallel prediction-planning as an effective end-to-end autonomous-driving design.
Takeaways & Limitations
Performance still falls behind single-task methods for some tasks, while dataset scale and open-loop evaluation limit comprehensive assessment.
Abstract
from arXiv · showhide
The well-established modular autonomous driving system is decoupled into different standalone tasks, e.g. perception, prediction and planning, suffering from information loss and error accumulation across modules. In contrast, end-to-end paradigms unify multi-tasks into a fully differentiable framework, allowing for optimization in a planning-oriented spirit. Despite the great potential of end-to-end paradigms, both the performance and efficiency of existing methods are not satisfactory, particularly in terms of planning safety. We attribute this to the computationally expensive BEV (bird's eye view) features and the straightforward design for prediction and planning. To this end, we explore the sparse representation and review the task design for end-to-end autonomous driving, proposing a new paradigm named SparseDrive. Concretely, SparseDrive consists of a symmetric sparse perception module and a parallel motion planner. The sparse perception module unifies detection, tracking and online mapping with a symmetric model architecture, learning a fully sparse representation of the driving scene. For motion prediction and planning, we review the great similarity between these two tasks, leading to a parallel design for motion planner. Based on this parallel design, which models planning as a multi-modal problem, we propose a hierarchical planning selection strategy , which incorporates a collision-aware rescore module, to select a rational and safe trajectory as the final planning output. With such effective designs, SparseDrive surpasses previous state-of-the-arts by a large margin in performance of all tasks, while achieving much higher training and inference efficiency. Code will be avaliable at https://github.com/swc-17/SparseDrive for facilitating future research.
1 Introduction
SparseDrive addresses limitations of modular and existing end-to-end driving systems with a sparse scene representation and redesigned prediction-planning architecture. It reports improved task performance, planning safety, and efficiency over prior methods.
- Modular driving pipelines lose information and accumulate errors across sequential perception, prediction, and planning modules.
- Existing end-to-end methods rely on expensive BEV features and straightforward prediction-planning designs, limiting performance and efficiency.
- SparseDrive unifies detection, tracking, and online mapping through symmetric sparse perception and represents the scene with sparse instances.
- Its parallel motion planner simultaneously predicts multi-modal trajectories and uses hierarchical selection with collision-aware rescoring to choose the final trajectory.
- 19.4% lower average L2 error (0.58m vs. 0.72m) and 71.4% lower collision rate (0.06% vs. 0.21%) are reported for SparseDrive-B.
- SparseDrive surpasses prior methods across all metrics on nuScenes, especially collision rate, while achieving higher training and inference efficiency.
2 Related Work
Related work covers sparse and BEV-based 3D detection, tracking, online mapping, end-to-end motion prediction, and end-to-end planning. These lines of research progressively integrate driving tasks while addressing representation, association, interpretability, or trajectory-selection challenges.
- 2.1 Multi-view 3D Detection: Multi-view 3D detection methods range from BEV feature lifting and predefined queries to implicit view transformation and explicit sparse 3D anchors.
- 2.2 Multi-object Tracking: Most multi-object tracking systems use tracking-by-detection with post-processing, while track-query methods model tracked instances in a streaming manner.
- 2.3 Online Mapping: Online mapping replaces costly HD maps with learned map-instance construction using semantic segmentation, autoregressive transformers, or point-set representations.
- 2.4 End-to-End Motion Prediction: End-to-end motion prediction seeks to avoid cascading pipeline errors by jointly predicting current and future boxes, behaviors, or trajectories.
- 2.5 End-to-End Planning: End-to-end planning has used direct trajectory prediction, explicit cost maps, or unified queries, balancing optimization difficulty, interpretability, and trajectory selection.
3 Method
SparseDrive combines symmetric sparse perception with a parallel motion planner to represent the scene sparsely, jointly predict motion, and select a safe trajectory. Its end-to-end training combines task losses while preserving efficiency-oriented sparse processing.
- 3.1 Overview: SparseDrive encodes multi-view features into sparse surrounding-agent and map instances, then feeds them with an initialized ego instance to the motion planner.The framework contains an image encoder, symmetric sparse perception, and parallel motion planner.
- 3.2 Symmetric Sparse Perception: Symmetric sparse perception unifies detection, tracking, and online mapping through structurally shared branches for agent and map-element instances.Detection uses instance features and anchor boxes, while mapping represents static elements with anchor polylines; tracking assigns persistent IDs after confidence exceeds a threshold.
- 3.3 Parallel Motion Planner: The ego instance is initialized from front-camera features and known ego geometry, while an auxiliary status task avoids initializing velocity from ground truth.The front-camera feature map supplies semantic context and dense complementary information for obstacles that sparse perception may miss.
- 3.3 Parallel Motion Planner: Agent-level spatial-temporal interactions combine ego and surrounding instances through temporal, agent-agent, and agent-map attention before trajectory prediction.An instance memory queue stores (N_d + 1) × H frames, and instance-level temporal attention focuses each instance on its own history.
- 3.3 Parallel Motion Planner: The planner predicts multi-modal trajectories for surrounding agents and the ego vehicle simultaneously, then hierarchically selects a command-consistent trajectory using collision-aware rescoring.Trajectories are scored across motion-prediction and planning modes; proposals with high collision probability receive reduced scores.
- 3.4 End-to-End Learning: End-to-end learning uses detection, mapping, motion, planning, and depth losses, with sparse perception trained first and both modules jointly optimized afterward.The overall loss is L = L_det + L_map + L_motion + L_plan + L_depth; stage two trains the sparse perception module and planner without frozen weights.
4 Experiments
On nuScenes, SparseDrive outperforms prior methods across perception, prediction, planning, and efficiency evaluations. Ablations support its parallel, multi-modal planner and collision-aware rescore design.
- Main Results: SparseDrive-S surpasses previous state-of-the-art methods across all tasks, while SparseDrive-B further advances performance boundaries.Experiments use the nuScenes dataset and compare against modularized and end-to-end methods.
- Perception: 49.6% mAP and 58.8% NDS improve 3D detection over UniAD by +11.6% mAP and +9.0% NDS.Tracking reaches 50.1% AMOTA with 632 identity switches, while online mapping reaches 56.2% mAP.
- Prediction: 0.60m minADE and 0.96m minFDE give SparseDrive the best reported motion-prediction performance, reducing UniAD errors by 15.5% and 5.9%.The model also reports 13.2% MissRate and 0.555 EPA.
- Planning: 0.58m L2 error and 0.06% collision rate are the lowest reported planning values, reducing error and collisions versus VAD by 19.4% and 71.4%.The collision rate is safety-critical in the reported comparison.
- Efficiency: 7.2× faster training and 5.0× faster inference are achieved by SparseDrive-S, while the base model is 4.8× and 4.1× faster than UniAD.The efficiency comparison uses different GPUs for UniAD and SparseDrive, as specified in the table caption.
- Ablation Study: Collision-aware rescore reduces collision rate from 0.12% to 0.08% with negligible L2-error increase, unlike the compared post-optimization strategy.The authors report that post-optimization degraded L2 error and did not make planning safer under their reimplemented metric.
- Ablation Study: Increasing planning modes improves performance continuously until saturation at 6 modes, supporting the multi-modal planning design.The planner ablations also examine parallel prediction and planning, ego-instance initialization, and agent-temporal cross-attention.
5 Conclusion and Future Work
SparseDrive combines sparse scene representation with revised task design to achieve strong performance and efficiency. The authors identify remaining gaps in single-task performance, dataset scale, and open-loop evaluation.
- Conclusion: SparseDrive explores sparse scene representation and revises end-to-end task design, achieving remarkable performance and high efficiency.The authors hope these results encourage reconsideration of task design in end-to-end autonomous driving.
- Future Work: The end-to-end model still trails single-task methods on some tasks, including online mapping.The authors also cite insufficient dataset scale and the limited comprehensiveness of open-loop evaluation.
A Metrics
The appendix defines task-specific evaluation metrics for detection, tracking, online mapping, and planning. Planning uses L2 error and collision rate, with a revised collision-rate implementation addressing stated drawbacks in prior implementations.
- Perception Metrics: Detection uses mAP, mATE, mASE, mAOE, mAVE, mAAE, and NDS; tracking uses AMOTA, AMOTP, RECALL, and IDS.These evaluations follow standard protocols.
- Planning Metrics: Planning is evaluated with L2 error and collision rate, while prior collision-rate implementations are described as inaccurate because of two drawbacks.The L2-error evaluation is aligned with VAD.
B.1 Perception
The sparse perception module uses a six-layer decoder comprising one non-temporal and five temporal decoders, alongside sparse anchors and bounded perception ranges.
- Decoder: The sparse perception decoder has 6 layers: 1 non-temporal decoder and 5 temporal decoders.The implementation also uses 900 anchor boxes, 100 anchor polylines, and 20 points per map element.
- Anchors and Ranges: Anchor locations are obtained by K-Means clustering on the training set, while other anchor-box parameters use fixed initialization values.Detection and online mapping use different perception ranges, and Flash Attention reduces GPU memory use.
B.2 Motion Planner
The motion planner uses multiple learned trajectory modes and decodes ego status from the ego feature. A collision-aware rescore module uses confident predicted trajectories to assess potential collisions.
- Motion Planner: The planner uses 6 motion-prediction modes and 6 planning modes, with 12 and 6 future timestamps respectively.The instance memory queue stores 3 frames.
- Motion Planner: After spatial-temporal interactions, an MLP decodes current ego status from the ego feature.
- Motion Planner: K-Means clustering transforms prior intention points into motion and planning mode queries for multi-modal trajectory prediction and planning.The queries are combined with agent instance features before trajectory and score decoding.
- Motion Planner: The collision-aware rescore module uses the two most confident predicted trajectories to determine whether the ego vehicle will collide with surrounding obstacles.
B.3 Loss Functions
The training objective combines perception, motion-prediction, planning, and depth-estimation losses. Matching, classification, regression, and winner-takes-all procedures define how these task losses are computed.
- Loss Functions: Perception matches each ground truth with one prediction using the Hungarian algorithm.
- Loss Functions: Detection loss combines Focal classification loss with L1 box-regression loss, while online mapping uses a similar loss.
- Loss Functions: Depth estimation uses L1 regression loss, with loss weights λdet_cls = 2, λdet_reg = 0.25, λmap_cls = 1, λmap_reg = 10, and λdepth = 0.2.
- Loss Functions: For motion prediction and planning, the lowest-ADE trajectory is positive and the others are negative under a winner-takes-all strategy.Planning additionally predicts ego status and uses Focal classification and L1 regression losses.
- Loss Functions: Motion and planning loss weights are λmotion_cls = 0.2, λmotion_reg = 0.2, λplan_cls = 0.5, λplan_reg = 1.0, and λplan_status = 1.0.
B.4 Training Details
Training uses AdamW with cosine annealing, while visualizations show SparseDrive learning different intersection-turning modes and responding to moving agents or obstacles.
- Training Details: SparseDrive is trained with the AdamW optimizer and a Cosine Annealing scheduler.The training hyperparameters are listed in Table 7.
- Training Details: Visualization results show SparseDrive learning different turning modes at intersections.
- Training Details: Visualization results show SparseDrive learning to yield to moving agents or avoid collision with obstacles.