PaperlayerView exact revision on arXiv

Source-linked AI summary

SparseAD: Sparse Query-Centric Paradigm for Efficient End-to-End Autonomous Driving

Diankun Zhang, Guoan Wang, Runwen Zhu, Jianbo Zhao, Xiwu Chen, Siyu Zhang, Jiahao Gong, Qibin Zhou, Wenyuan Zhang, Ningzi Wang, Feiyang Tan, Hangning Zhou, Ziyao Xu, Haotian Yao, Chi Zhang, Xiaojun Liu, Xiaoguang Di, Bin Li

arXiv:2404.06892v1cs.CV

TL;DR

End-to-end autonomous driving methods still lag single-task methods on subtasks, while dense BEV features make extending them to more modalities or tasks costly. SparseAD represents the driving scenario with sparse queries and unifies perception with motion planning. On nuScenes, it achieves SOTA full-task performance among end-to-end methods and narrows the gap to single-task methods.

  • Problem

    Dense BEV-based end-to-end methods lag corresponding single-task methods on subtasks, and dense features increase computation, memory, and deployment burden.

    Source: §1 · p. 2

  • Method

    SparseAD represents the scenario across space and time with sparse queries and combines sparse perception for detection, tracking, and online mapping with a motion planner.

    Source: §1 · p. 3

  • Results

    SparseAD achieves SOTA full-task performance among end-to-end methods on nuScenes and significantly narrows the performance gap to single-task methods.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The nuScenes experiments demonstrate SparseAD's effectiveness and superiority, and the authors see potential for end-to-end driving in performance, efficiency, and scalability.

    Source: §6 · p. 15

    Balancing multiple tasks during optimization remains an open issue.

    Source: §6 · p. 15

Abstract

from arXiv · show

End-to-End paradigms use a unified framework to implement multi-tasks in an autonomous driving system. Despite simplicity and clarity, the performance of end-to-end autonomous driving methods on sub-tasks is still far behind the single-task methods. Meanwhile, the widely used dense BEV features in previous end-to-end methods make it costly to extend to more modalities or tasks. In this paper, we propose a Sparse query-centric paradigm for end-to-end Autonomous Driving (SparseAD), where the sparse queries completely represent the whole driving scenario across space, time and tasks without any dense BEV representation. Concretely, we design a unified sparse architecture for perception tasks including detection, tracking, and online mapping. Moreover, we revisit motion prediction and planning, and devise a more justifiable motion planner framework. On the challenging nuScenes dataset, SparseAD achieves SOTA full-task performance among end-to-end methods and significantly narrows the performance gap between end-to-end paradigms and single-task methods. Codes will be released soon.

1 Introduction

The introduction presents SparseAD as a sparse query-centric alternative to dense BEV-based end-to-end driving, motivated by subtask performance gaps and computational costs. It reports stronger full-task end-to-end performance and a narrower gap to single-task methods.

1 Introduction

  • Earlier end-to-end approaches either predict plans directly from raw sensors, with reported shortcomings in optimization, explainability, and planning performance, or integrate multiple tasks modularly.Source: §1 · p. 2
  • Prior dense BEV-centric end-to-end methods lag single-task methods on subtasks and increase computation, memory use, and deployment burden.Source: §1 · pp. 2–3
  • SparseAD represents the driving scenario across space and time with sparse queries, without dense BEV features.Source: §1 · p. 3
  • The architecture combines sparse perception for detection, tracking, and online mapping with a motion planner for prediction and planning.Source: §1 · p. 3
  • SparseAD achieves SOTA performance among end-to-end methods on nuScenes and significantly narrows the performance gap between end-to-end and single-task methods.Source: §1 · p. 3

2 Related Work

Related work covers sparse and dense approaches to perception, online mapping, motion prediction, and planning. It describes limits of existing methods in task optimization, computational cost, and real-world performance.

2.1 Detection and Tracking

  • Camera-based 3D perception includes dense BEV methods and sparse-query methods, while joint detection and tracking can suffer query-optimization imbalance that degrades detection.Source: §2.1 · pp. 3–4

2.2 Online Mapping

  • Prior online mapping methods use dense BEV features, whose computational cost grows with perception distance and complicates integration into end-to-end models.Source: §2.2 · p. 4

2.3 Motion Prediction

  • Motion prediction research spans rasterized, vectorized, graph-based, and more efficient environmental representations, with some works applying trajectory prediction end to end.Source: §2.3 · pp. 4–5

2.4 Planning

  • Planning approaches include optimization, search, sampling, and learning-based methods, while simulation-to-real gaps can limit real-world performance; recent end-to-end work includes dense BEV and vectorized approaches.Source: §2.4 · p. 5

3 Method

SparseAD uses sparse queries as a representation of the driving scenario across tasks, with no dense BEV features. Its sparse perception and motion planner use temporal information and driving constraints to produce perception outputs and final plans.

3.1 Overview

  • Sparse queries represent the driving scenario across modules and carry information and gradients for end-to-end multi-task optimization without dense BEV features.Source: §3.1 · pp. 5–6

3.2 Sparse Perception

  • Two same-structure temporal decoders unify detection, tracking, and online mapping, with separate heads producing obstacle and map outputs.Source: §3.2 · pp. 6–7
  • Scene-level memory retains queries without cross-frame associations, while instance-level memory preserves associations between tracked obstacles across frames.Source: §3.2 · p. 7
  • Map queries interact with sensor tokens and high-confidence historical map queries, then are updated for future frames; the authors identify this as sparse online mapping within a unified perception architecture.Source: §3.2 · pp. 7–8

3.3 Motion Planner

  • The motion predictor jointly predicts trajectories for surrounding agents and the ego vehicle, using ego predictions as motion priors for subsequent planning.Source: §3.3 · pp. 8–9
  • The planning optimizer considers high-level commands, safety, and kinematic requirements, including decoded lateral and longitudinal relationships between agents and the ego vehicle.Source: §3.3 · pp. 8–9
  • Motion queries aggregate information from track and map queries and model interactions between surrounding agents and the ego vehicle over future scenes.Source: §3.3 · pp. 8–9
  • An auxiliary kinematic loss supervises ego velocity, acceleration, and yaw decoded from the ego query.Source: §3.3 · p. 10

4 Experiments

On nuScenes, SparseAD reports stronger full-stack performance than UniAD [7] with nearly 1× faster speed and 35% memory footprint. The experiments also report task-specific results, ablations, and qualitative performance in a challenging driving scenario.

4 Experiments

  • Experiments evaluate image-only SparseAD-B and SparseAD-L and image-plus-radar SparseAD-BR on nuScenes, using V2-99 for B and BR and ViT-Large for L.Input image resolutions are 1600 × 640 for B and BR and 1600 × 800 for L.Source: §4 · p. 10

4.1 Comprehensive Results

  • Across full-stack tasks, SparseAD outperforms UniAD [7] at nearly 1× faster speed and 35% memory footprint; radar and ViT-Large further improve performance at almost negligible extra costs.The paper reports that scaling to ViT-Large further improves comprehensive performance.Source: §4.1 · pp. 10–11

4.2 Multi-Task results

  • SparseAD-Large achieves 53.6% mAP, 62.5% NDS, and 60.6% AMOTA, outperforming Sparse4Dv3 [39] overall, while camera-only results reach 47.5% mAP, 57.8% NDS, 53% AMOTA, and 60.8% Recall.The paper also reports that SparseAD-B outperforms most popular detection-only, tracking-only, and end-to-end tracking methods while remaining comparable to StreamPETR [10] and QTrack [83].Source: §4.1 · p. 10, §4.2 · p. 11
  • SparseAD achieves 34.2% mAP for sparse online mapping, outperforming most prior methods but lagging StreamMapNet [53]; boundary evaluation separates road segment and lane over a 102.4m × 102.4m range.The authors identify effective use of other modalities such as radar as an area for further exploration.Source: §4.2 · p. 12, Table 3 · p. 12
  • SparseAD attains 0.83m minADE, 1.58m minFDE, 18.7% Miss Rate, and 0.308 EPA, the best reported end-to-end motion-prediction performance; SparseAD-B also achieves the lowest average L2 error and Collision Rate.Planning performance improves further with radar or more powerful backbone networks.Source: §4.1 · p. 11, §4.2 · pp. 12–13

4.3 Ablations

  • Scene-level memory improves detection and mapping, while tracking initially degrades slightly; two-level memory updates and track-query augmentation further improve all perception tasks.The authors attribute the initial tracking degradation to potentially unstable supervision across different queries during training.Source: §4.3 · p. 13, Table 5 · p. 13
  • Planning performance declines to varying degrees when motion-predictor strategies or planning-optimizer constraints are removed; ego status is excluded to prevent shortcut learning in open-loop planning.The ablations cover motion-query initialization, agent-memory interaction, ego kinematic constraints, and ego-agent relationship constraints.Source: §4.3 · p. 13, Table 6 · p. 14

4.4 Qualitative Results

  • In a challenging scene with nearby front and rear agents, roadside obstacles, potential occlusions, and irregular boundaries, SparseAD produces accurate perception and a reasonable, safe planning trajectory.The visualization displays predictions across surround-view images and BEV; LiDAR appears only for visualization and is not a model input.Source: §4.4 · p. 14, Figure 5 · p. 14

5 Discussion

The discussion identifies shortcomings in existing collision-rate calculations and proposes a more precise and sensible calculation method.

5 Discussion

  • Collision rate is identified as a key metric for evaluating end-to-end autonomous driving methods.The discussion focuses on shortcomings in how the metric is calculated.Source: §5 · p. 14
  • The authors identify serious disadvantages in existing collision-rate calculation methods.They discuss the issue in Appendix F.Source: §5 · pp. 14–15
  • The paper proposes a new collision-rate calculation method described as more precise and sensible.The authors locate its detailed discussion in Appendix F.Source: §5 · pp. 14–15

6 Conclusion and Future Work

SparseAD condenses the driving scenario into sparse queries and uses an architecture of sparse perception and a motion planner; experiments on nuScenes demonstrate its effectiveness. The authors identify task-balancing and sparse online-mapping performance as areas for future work.

6 Conclusion and Future Work

  • SparseAD condenses spatial and temporal driving-scenario information into sparse queries and uses a succinct architecture of sparse perception and a motion planner.The authors report substantial experiments on nuScenes demonstrating the method’s effectiveness and superiority.Source: §6 · p. 15
  • Balancing multiple tasks during optimization remains an open issue, and further improving sparse online-mapping performance is a future research direction.Source: §6 · p. 15

A Overview

  • The supplementary material covers task definitions, implementation details, further experiments, a SparseAD variant, collision-rate evaluation, and qualitative visualizations.Its variant explores the trade-off between perception precision and recall and temporal consistency; it also proposes a sparse-based collision-rate calculation method.Source: A Overview · p. 22

B Definition of Each Task

The paper describes four autonomous-driving tasks: detection and tracking of obstacles, online modeling of road structure and semantics, prediction of traffic participants’ future states, and planning a suitable ego-vehicle trajectory.

Detection and Tracking

Online Mapping

  • Online mapping models driving-environment structure and semantics as sparse line segments representing categorized road elements.Examples include dividers, pedestrian crosswalks, road boundaries, and lane lines; the task aims to replace offline annotated HD maps.Source: B Definition of Each Task · p. 22

Motion Prediction

  • Motion prediction forecasts possible future states of traffic participants to assist ego-vehicle decision-making, using historical information and maps.Source: B Definition of Each Task · p. 22

Planning

  • Planning generates an ego-vehicle trajectory that meets safety, comfort, and kinematic requirements while considering current and possible future states from upstream tasks.The target trajectory is collision-free, comfortable, and controllable.Source: B Definition of Each Task · p. 23

C Implementation details

This section outlines SparseAD’s implementation, covering multimodal sensor encoding and sparse temporal-query components for perception, motion prediction, and ego planning, along with planning constraints, temporal memory, nuScenes and model configurations, training and inference procedures, and evaluation metrics.

C.1 Problem Definition

  • Camera images and optional radar or LiDAR point clouds are encoded and concatenated as sensor tokens, while learnable and memory queries represent task priors and temporal information.Memory queries propagate information across frames and convey upstream information to downstream tasks.Source: §C.1 · p. 23

C.2 Sparse Perception

  • Two Temporal Decoders use sensor tokens and task-specific queries for detection and tracking and online mapping; heads decode 3D boxes and sparse map elements as Bezier curves.The Bezier map head regresses curve parameters and uniformly samples points along each curve for map-element locations.Source: §C.2 · pp. 24–25

C.3 MotionPlanner

  • Motion and ego queries combine encoded history with shared and ego-specific embeddings, and newborn instances receive dummy history information.The planner encodes agent and ego historical positions with a shared MLP and aggregates instance history perception information using cross-attention.Source: §C.3 · p. 26
  • The predictor uses K-means trajectory anchors and agent-memory, agent-map, and agent-social interactions, with temporal, curve, and BEV positional encodings for trajectory alignment and scene locations.During training, it supervises the multimodal trajectory closest to ground truth; the Planning Optimizer considers safety and kinematic constraints when generating the ego trajectory.Source: §C.3 · pp. 26–28
  • The Planning Optimizer enforces Ego-Map, Ego-Agent, and Ego-Kinematic constraints using reciprocal future-trajectory constraints in longitudinal and lateral directions, consistent with [12].The constraints relate trajectories of other agents and the ego vehicle.Source: §C.3 · p. 28

C.4 Frame Summary

  • EMMB combines frame-level scene memory with instance-level memory to retain global and per-instance temporal information.Scene memory uses FIFO storage for the previous M frames; instance memory uses FIFO for each instance without a lifetime limit, and is removed after consecutive missed detections. It stores task queries and decoded outputs, applies ego transformation to positional information, and converts high-confidence detection queries into ID-assigned track queries.Source: §C.4 · pp. 28–29

D.1 Datasets

  • nuScenes provides 700/150/150 training, validation, and test scenes, with roughly 40 annotated key frames per 20-second scene at 2 Hz.Each sample contains six cameras covering 360° horizontally, plus LiDAR and radar point clouds [19].Source: §D.1 · pp. 29–30

D.2 Configuration Details

  • SparseAD-Base uses VoVNetv2 and 1600 × 640 images, while SparseAD-Large uses ViT-Large and 1600 × 800 images; both operate over a [−51.2m, 51.2m] range on each axis.EMMB history is set to M = 5 (about 2.5 seconds). Detection/tracking uses 644 learnable and 256 scene-memory detection queries, propagated high-score track queries, 0.5 track-query drop probability, and 0.2 approximate-negative sampling probability. Mapping uses [35] [30] [15] [25] learnable and [14] [12] [6] [10] memory queries for Divider, Cross, Road Segment, and Lane, respectively, with type-specific Bezier piece/degree settings.Source: §D.2 · p. 30
  • The motion planner uses five history frames, predicts other agents 12 frames ahead and ego motion six frames ahead, and assigns six multimodal motion queries to each agent.K-means on nuScenes training data supplies prior intentions; training supervises only the prediction closest to ground truth, and ego predictions inform the Planning Optimizer. Training proceeds through perception, memory-enhanced tracking, and planner stages; the first two may be merged with slight performance degradation. Training uses AdamW and cosine annealing on eight A100 80GB GPUs; SparseAD-Base requires at most 17.5GB, and undetected instances are removed after eight consecutive frames at inference.Source: §D.2 · p. 30, Training and Inference · pp. 30–31

D.3 Metrics

  • Sparse perception is evaluated with detection NDS and mAP, tracking AMOTA, AMOTP, RECALL, and IDS, and per-category mapping mAP at TP thresholds of 0.5m, 1m, and 2m.Motion prediction uses minADE, minFDE, MR, and EPA; planning uses L2 error and ColRate, with collision rate recalculated using a sparse method. The motion benchmark follows VIP3D [63].Source: Metrics for Sparse Perception · p. 31, Metrics for Motion Planner · pp. 31–32

D.4 Supplementary Experiments

On the nuScenes test set, SparseAD-Large slightly exceeds a cited state-of-the-art method on detection and tracking, while SparseAD-Base remains inferior to state-of-the-art single-task methods. End-to-end gradient optimization slightly improves motion prediction and planning but slightly degrades perception; the authors note that a fully fair comparison and test-set results for other tasks are unavailable.

Evaluation on Test Set

  • 56.6% mAP, 64.8% NDS, and 58.8% AMOTA: SparseAD-Large is slightly better than state-of-the-art method [39] on the nuScenes test set.SparseAD-Base is comparable to popular methods [25] [35] [37] [84] but remains inferior to state-of-the-art single-task methods [10] [39].Source: Evaluation on Test Set · p. 32
  • A fully fair test-set comparison is unavailable because UniAD was not evaluated there; only detection and tracking can be fairly compared on the nuScenes official benchmark.The paper provides no test-set evaluation for mapping, motion prediction, or planning.Source: Evaluation on Test Set · p. 32

Module runtime

  • The module-runtime measurements cover SparseAD-Base and SparseAD-Large on one NVIDIA RTX A100 80GB GPU.Source: Table 9 · p. 33

End-to-End Gradient Optimization

  • End-to-end gradient optimization slightly improves motion prediction and planning while slightly degrading perception, a trade-off the authors consider acceptable for planning accuracy and safety.The extra memory footprint is described as acceptable for academic-level computing power, such as an NVIDIA RTX 3090 GPU.Source: End-to-End Gradient Optimization · p. 33

D.5 Runtime Analysis

SparseAD-Base and SparseAD-Large differ substantially in runtime, with Sparse Perception the largest Base module and the ViT-Large backbone dominating Large latency. The appendix also motivates tracking with scene-level detection memory queries and describes how they replace track queries.

D.5 Runtime Analysis

  • SparseAD-Base runs at 287ms, with Sparse Perception accounting for 36.2% of latency; SparseAD-Large runs at 1386ms, with ViT-Large accounting for 83.1%.Maintaining EMMB takes 23ms, about 8.1% of SparseAD-Base latency; the other SparseAD-Large modules rise only slightly in latency compared with Base.Source: §D.5 · pp. 33–34

E.1 Motivation And Method

  • Scene-level detection memory queries may serve as track queries because they are not bound to ground truth through ID relationships during training, avoiding the supervision imbalance affecting track queries.The authors propose this as a way to achieve end-to-end tracking without declining detection performance.Source: Motivation · p. 34
  • Training discards instance-level track queries and changes the learnable-to-memory detection query ratio from 644:256 to 200:700.Detection queries matched to the same ground truth are concatenated into instances; at inference, memory queries inherit prior-frame IDs, while learnable queries receive new IDs.Source: Motivation · p. 34

E.2 Validation Experiments

On nuScenes validation, removing hard association improves SparseAD's perception metrics, while increasing IDS. Prediction and planning performance remain comparable between the two variants.

E.2 Validation Experiments

  • +1.2% NDS, +2.4% mAP, and +1.7% AMOTA for SparseAD without hard association versus SparseAD-Base; with ViT-Large, the increases are +2.3% NDS, +3.7% mAP, and +2.3% AMOTA.Table 10 reports the comparison of sparse perception performance under hard-association supervision.Source: §E.2 · p. 34, Table 10 · p. 35
  • SparseAD without hard association also increases IDS, while its prediction and planning capabilities are comparable to the hard-association variant.The authors attribute the comparable downstream performance to excessive IDS and insufficient trajectory consistency despite recalling more challenging samples.Source: §E.2 · p. 35, Table 11 · p. 35

F Discussion about the Collision Rate Metric

The authors argue that heading-agnostic boxes and coarse occupancy grids make conventional collision-rate evaluation unreliable. Their sparse-based metric changes the ground-truth collision estimate substantially and is also used to reassess planning results.

F Discussion about the Collision Rate Metric

  • The conventional calculation can misjudge collisions by ignoring ego-vehicle yaw variation and using occupancy grids with about 0.5-meter resolution.The authors say such grid resolution can exceed the possible minimum safe distance between vehicles; Fig. 7 illustrates both sources of error.Source: F Discussion about the Collision Rate… · pp. 35–36, Figure 7 · p. 37
  • The conventional method estimates over 10% collision for ground-truth trajectories, which the authors consider unrealistic even accounting for possible IMU error.Previous benchmarks excluded ground-truth trajectories marked as collisions; the authors attribute the inflated estimate to coarse grids and neglected yaw variation.Source: F Discussion about the Collision Rate… · p. 36
  • The sparse-based method derives obstacle positions and headings from nuScenes, approximates ego heading from adjacent planned-trajectory positions, and checks bounding-box overlap without occupancy grids.The collision rate is computed only when the ego-vehicle box overlaps another obstacle's box.Source: F Discussion about the Collision Rate… · pp. 36–37
  • 0.58%: the proposed sparse-based metric reports this collision rate for ground-truth ego trajectories, nearly 20x below the coarse occupancy-grid result.Table 12 compares the two calculation methods on ground-truth future trajectories.Source: Table 12 · p. 36, F Discussion about the Collision Rate… · p. 37
  • Re-evaluation with the proposed metric shows a collision-rate degradation for UniAD relative to its paper's reported results, as expected with hard cases included.Table 13 compares planning results from different methods under the sparse-based metric; UniAD's reproduced results are marked with ‡.Source: Table 13 · p. 36, F Discussion about the Collision Rate… · p. 37

G Qualitative results on Various Cases

The qualitative examples show SparseAD planning across intersection, turning, and nighttime scenes, including pedestrian and vehicle avoidance. They also expose a failure involving underuse of prior ego-vehicle state.

G Qualitative results on Various Cases

  • The visualizations cover intersection, turning, and nighttime scenes, with multiple camera views and a BEV result for each scenario.Fig. 8 presents these examples as qualitative planning results.Source: Figure 8 · p. 38
  • Despite straight-going scenarios dominating the nuScenes training set, the intersection examples show command-conditioned decisions and predicted trajectories for other vehicles.Fig. 9 describes successful completion of the intersection decision-making task.Source: Figure 9 · p. 39
  • In the avoidance examples, SparseAD adjusts its trajectory around a distant pedestrian and decelerates or stops for lateral crossings at intersections.Fig. 10 shows three obstacle-avoidance scenarios.Source: Figure 10 · p. 40
  • A failure example plans straight while the ego-vehicle remains stationary; the authors say this choice is not unreasonable from perception but reflects underuse of prior ego states.The resulting plan differs from ground truth.Source: Figure 11 · p. 41

Related papers

Loading 2404.06892v1…