PaperlayerView exact revision on arXiv

Source-linked AI summary

Planning-oriented Autonomous Driving

Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, Hongyang Li

arXiv:2212.10156v2cs.CVcs.RO

TL;DR

Autonomous-driving approaches may isolate tasks or omit components from planning-oriented systems. UniAD unifies driving tasks through query-based interfaces and outperforms naive multi-task learning across essential metrics, including a -0.15m avg.L2 delta.

  • Problem

    Prior autonomous-driving approaches either isolate tasks or do not consistently include components needed for a planning-oriented framework.

    Source: §1 · pp. 1–2

  • Method

    UniAD is a planning-oriented end-to-end framework that connects driving-task modules through query-based interfaces to coordinate information toward planning.

    Source: §1 · p. 2

  • Results

    -0.15m avg.L2 versus naive multi-task learning, with UniAD outperforming it across all essential metrics.

    Source: §3.1 · p. 6

  • Takeaways & Limitations

    The ablations indicate that both motion forecasting and occupancy prediction are required for the evaluated safe-planning objective.

    Source: §3.1 · p. 6

    Coordinating UniAD's multiple tasks requires extensive computational power, particularly when trained with temporal history.

    Source: §4 · p. 8

Abstract

from arXiv · show

Modern autonomous driving system is characterized as modular tasks in sequential order, i.e., perception, prediction, and planning. In order to perform a wide diversity of tasks and achieve advanced-level intelligence, contemporary approaches either deploy standalone models for individual tasks, or design a multi-task paradigm with separate heads. However, they might suffer from accumulative errors or deficient task coordination. Instead, we argue that a favorable framework should be devised and optimized in pursuit of the ultimate goal, i.e., planning of the self-driving car. Oriented at this, we revisit the key components within perception and prediction, and prioritize the tasks such that all these tasks contribute to planning. We introduce Unified Autonomous Driving (UniAD), a comprehensive framework up-to-date that incorporates full-stack driving tasks in one network. It is exquisitely devised to leverage advantages of each module, and provide complementary feature abstractions for agent interaction from a global perspective. Tasks are communicated with unified query interfaces to facilitate each other toward planning. We instantiate UniAD on the challenging nuScenes benchmark. With extensive ablations, the effectiveness of using such a philosophy is proven by substantially outperforming previous state-of-the-arts in all aspects. Code and models are public.

1. Introduction

The introduction argues that autonomous-driving tasks should be coordinated around planning rather than isolated or merely grouped under a shared backbone. It presents UniAD as a planning-oriented, end-to-end framework connecting five essential tasks through queries.

1. Introduction

  • Standalone models risk information loss, error accumulation, and feature misalignment, while shared-backbone multi-task learning may cause negative transfer.The introduction notes that multi-task learning can share feature abstractions and save onboard computation, but may transfer knowledge undesirably across tasks.Source: §1 · pp. 1–2
  • End-to-end systems should select and prioritize preceding tasks in favor of planning, avoiding standalone errors and multi-task negative transfer.The paper argues for a planning-oriented design with components chosen to help planning rather than an indiscriminate stack of tasks.Source: §1 · p. 2
  • Direct trajectory prediction without perception or prediction supervision is called inadequate for safety and interpretability in dynamic urban settings; prior explicit pipelines may omit planning-relevant components [11] [30] [38] [57] [82].The introduction contrasts these designs with a system that organizes preceding tasks to facilitate planning.Source: §1 · p. 2
  • UniAD connects five essential driving tasks with queries that soften upstream error accumulation and flexibly encode interactions among agents.The framework is intended to support a safe and robust system and facilitate cooperation among perception, prediction, and planning tasks.Source: §1 · p. 2
  • The authors report that benchmark instantiation and extensive ablations verify UniAD's superiority over previous state-of-the-art methods in all aspects.The introduction identifies the evaluation as using a challenging benchmark for realistic scenarios.Source: §1 · p. 2

2. Methodology

UniAD organizes perception, prediction, and planning as a query-connected, transformer-decoder pipeline optimized end to end toward ego-vehicle planning. Its modules share agent, map, motion, and occupancy representations so upstream scene understanding informs downstream predictions and planning.

2. Methodology

  • UniAD connects four perception and prediction modules to an ego-vehicle planner through task queries, with BEV features providing a shared scene representation.The planner uses the ego query and predicted occupancy to produce waypoints while avoiding occupied regions.Source: §2 · pp. 2–3, Figure 2 · p. 3

2.1. Perception: Tracking and Mapping

  • TrackFormer uses detection and persistent track queries to jointly detect and track agents, while MapFormer represents road elements as map queries for downstream motion forecasting.Track queries aggregate temporal information; MapFormer supervises outputs at each layer and forwards its final map queries to MotionFormer.Source: §2.1 · p. 3

2.2. Prediction: Motion Forecasting

  • MotionFormer predicts top-k multimodal trajectories for all agents in one scene-centric pass, combining agent-agent, agent-map, and agent-goal interactions.It also models ego-agent interactions; a non-linear smoother [7] adjusts training targets for imperfect upstream starting positions, with target optimization confined to training.Source: §2.2 · pp. 3–4

2.3. Prediction: Occupancy Prediction

  • OccFormer combines scene-level dense features with agent-level semantics to predict future occupancy and preserve each agent’s identity without heavy post-processing.It produces instance-level occupancy through matrix multiplication of dense scene and agent features.Source: §2.3 · pp. 4–5

2.4. Planning

  • The planner combines the ego query’s multimodal intentions with a learned navigation-command embedding, attends to BEV features, and decodes future waypoints.At inference, Newton’s method adjusts the predicted trajectory to stay near the original plan while avoiding occupied grids predicted by OccFormer.Source: §2.4 · p. 5

2.5. Learning

  • Table 2 reports that UniAD outperforms the naive multi-task-learning scheme on prediction and planning without substantial perceptual performance drop.Training first jointly optimizes tracking and mapping for six epochs, then trains all modules end to end for 20 epochs; tracking assignments are reused by motion and occupancy nodes.Source: §2.5 · p. 5, Table 2 · p. 6

3. Experiments

Experiments on nuScenes evaluate UniAD’s planning-oriented task coordination, individual task modules, qualitative behavior, and module-design ablations.

3.1. Joint Results

  • Exp.12 outperforms naive multi-task Exp.0 across all essential metrics, supporting the planning-oriented design.Reported gains are -15.2% minADE, -17.0% minFDE, -3.2 MR(%), +4.9 IoU-f.(%), +5.9 VPQ-f.(%), -0.15m avg.L2, and -0.51 avg.Col.(%).Source: §3.1 · p. 6

3.2. Modular Results

  • UniAD improves tracking AMOTA(%) by +6.5 versus MUTR3D [104] and +14.2 versus ViP3D [30], while achieving the lowest ID switch score.For mapping, it gains +7.4 IoU(%) over BEVFormer on lane segmentation, though it trails tracking-by-detection methods with complex associations and perception-oriented methods on specific classes.Source: §3.2 · p. 6, Table 3 · p. 7, Table 4 · p. 7
  • UniAD reduces motion-forecasting minADE by 38.3% versus PnPNet-vision [57] and 65.4% versus ViP3D [30].For occupancy prediction, it gains +4.0 and +2.0 IoU-near(%) versus FIERY [35] and BEVerse [105], respectively, with heavy augmentations.Source: Table 5 · p. 7, §3.2 · p. 7, Table 6 · p. 7
  • UniAD reduces planning L2 error and collision rate by 51.2% and 56.3%, respectively, versus ST-P3 on average across the planning horizon.It also notably outperforms several LiDAR-based counterparts.Source: Table 7 · p. 7, §3.2 · p. 7

3.4. Ablation Study

  • The rotated scene-level anchor boosts motion forecasting by -15.8% minADE, -11.2% minFDE, and +1.9 minFDE-mAP(%).Agent-goal interaction adds planning-oriented visual features to motion queries, while non-linear optimization accounts for perceptual uncertainty and improves metrics by -5.0% minADE, -8.4% minFDE, -1.0 MR(%), and +0.7 minFDE-mAP(%).Source: Table 8 · p. 7, §3.4 · pp. 7–8

3.3. Qualitative Results

  • UniAD’s qualitative results show consistent motion and occupancy predictions as the ego vehicle yields to a front car.Supplementary examples show later tasks can recover from some inaccurate earlier outputs, while failures mainly occur in long-tail scenarios such as large trucks and trailers.Source: §3.3 · p. 7, Figure 3 · p. 8

3.4. Ablation Study

  • Occupancy-guided attention masking improves nearby-area results by +1.0 IoU-n.(%) and +1.4 VPQ-n.(%), while reusing the mask feature further enhances performance.Without locality constraints, pixel-to-all-agent attention performs slightly worse than the attention-free baseline.Source: Table 9 · p. 8, §3.4 · p. 8
  • The planning ablations show that BEV attention, collision loss, and occupancy-based optimization all contribute, with collision rate reduced when all are applied.The comparison prioritizes lower collision rate for safety over naive trajectory mimicking measured by L2.Source: Table 10 · p. 8, §3.4 · p. 8

4. Conclusion and Future Work · Appendix

UniAD is a planning-oriented autonomous-driving framework that connects tasks through queries to support richer agent interaction. The conclusion reports that experiments verify the method, while identifying computational demands and system design as open challenges.

4. Conclusion and Future Work

  • UniAD proposes a planning-oriented pipeline and query-based connections among tasks to support richer representations for agent interaction.The work also analyzes the necessity of each perception and prediction module.Source: §4 · p. 8
  • Extensive experiments verify the proposed method in all aspects.The conclusion does not provide specific metrics or comparisons.Source: §4 · p. 8
  • Coordinating UniAD’s multiple tasks requires substantial computational power, especially when trained with temporal history.Developing and curating the system for lightweight deployment is proposed as future work.Source: §4 · p. 8
  • Future work includes whether to incorporate depth estimation and behavior prediction, and how to embed them in the system.Source: §4 · p. 8

A. Task Definition

UniAD defines a connected driving stack in which 3D detection and BEV mapping support motion forecasting and occupancy prediction, while planning uses upstream outputs to produce a safe ego trajectory.

Detection and tracking

  • 3D detection locates surrounding objects, and tracking associates them across frames with consistent IDs; the resulting 3D boxes and features QA pass to motion forecasting.A special ego-vehicle query regresses the ego location and is excluded from prediction-ground-truth matching.Source: A. Task Definition · p. 13

Online mapping

  • Online mapping segments lanes, drivable areas, dividers, and pedestrian crossings in BEV from onboard multi-view images, with map queries QM supporting agent-map interaction in forecasting.It substitutes for offline annotated HD maps.Source: A. Task Definition · p. 13

Motion forecasting

  • Motion forecasting uses encoded agent and map queries plus dense BEV features to predict K plausible, position-relative trajectories over T future timesteps.The ego query also predicts future ego-motion as a coarse planning estimate; its feature goes to the planner, while agent features encoding historical and future information go to occupancy prediction.Source: A. Task Definition · p. 13

Occupancy prediction

  • Occupancy prediction models dense future scene occupancy from observed BEV and agent features, merging instance-level maps by pixel-wise argmax to preserve agent identity [8].Unlike sparse-agent motion forecasting, it represents the whole scene over future timesteps.Source: A. Task Definition · p. 14

Planning

  • Planning predicts one ego trajectory from the ego query and dense BEV features, then optimizes it with predicted future occupancy to avoid collisions and ensure safety.This learning-based formulation contrasts with traditional rule-based planning conditioned on detection and prediction results.Source: A. Task Definition · p. 14

B. The Necessity of Each Task … D. Notations

The discussion motivates UniAD’s planning-oriented design by identifying complementary strengths and limitations in tracking, mapping, motion forecasting, and occupancy, then situates its joint perception, prediction, and planning approach among related work.

B. The Necessity of Each Task

  • UniAD combines agent-centric motion forecasting with whole-scene occupancy to support safe planning, since occupancy lacks agent identity and dynamics while dense features constrain its prediction horizon.Tracking can provide history for occluded agents, and HD maps can aid planning, but maps are costly to construct and may become outdated.Source: B. The Necessity of Each Task · p. 14

C.1. Joint perception and prediction

  • UniAD follows agent track queries [30] [104], adds an ego-vehicle query and online mapping, and applies nonlinear trajectory optimization to address inaccurate perception and HD-map risks or costs.Source: §C.1 · p. 14
  • Occupancy grids support collision-avoidance planning but lose agent identity and impose computational costs; UniAD uses agent-level information for occupancy prediction and unifies it with motion prediction.Prior approaches include future semantic occupancy distributions [82] and multi-view-camera occupancy prediction [35].Source: §C.1 · p. 14

C.2. Joint prediction and planning

  • UniAD encodes possible ego behaviors in scene-centric motion forecasting and uses an interpretable occupancy map to further optimize the plan for safety.Related approaches condition forecasting on ego goals [81] or presumed planning trajectories, or couple prediction with optimization [39] [40].Source: §C.2 · p. 15

C.3. End-to-end motion planning

  • Prior end-to-end planning methods use intermediate representations such as BEV semantics, smooth polynomial plans, contingency predictions, or cost volumes, but can risk inconsistent modules or rely on tailored costs and sampled trajectories.The passage cites BEV semantic predictions [19] [34] [83], polynomial planning [5], contingency planning [24], trajectory training [15], and cost volumes [94].Source: §C.3 · p. 15
  • UniAD uses ego-motion information without sophisticated cost design and jointly incorporates tracking with both prediction representations in an end-to-end model.Source: §C.3 · p. 15

D. Notations

E. Implementation Details … E.4. Occupancy Prediction

The implementation uses task-specific query interfaces and interaction modules to connect detection, tracking, mapping, motion forecasting, and occupancy prediction. It combines dense scene features with agent-level features to predict future occupancy.

E.1. Detection and Tracking

  • Detection and tracking build on BEVFormer’s BEV encoder and Deformable DETR head, adding MOTR-style track queries to follow previously observed instances by track ID.Track queries enable end-to-end tracking without heavy post-association.Source: §E.1 · p. 15
  • During training, detection queries use Hungarian matching, while track queries match ground-truth objects by track ID and retain predictions only above a 3D IoU threshold.At the first frame all queries detect newborn objects; from subsequent frames, detection queries handle newborn objects and track queries handle previously tracked objects. The retention threshold is 0.5 in practice.Source: §E.1 · p. 15
  • At inference, classification-score thresholds update queries, and a lifecycle mechanism removes a track only after its score stays below 0.35 for 2 seconds.The thresholds are 0.4 for detection queries and 0.35 for track queries, avoiding ground-truth-dependent filtering and preventing short occlusions from interrupting tracklets.Source: §E.1 · p. 15

E.2. Online Mapping

  • Online mapping separates instance-wise thing queries from a stuff query assigned to semantic elements such as drivable area.The thing queries model map instances including lanes, boundaries, and pedestrian crossings, and are bipartite-matched to ground truth; the stuff query uses class-fixed assignment. The setup uses 300 thing queries, one stuff query, six location decoder layers, and four mask decoder layers; thing queries after the location decoder serve as downstream map queries.Source: §E.2 · pp. 15, 17

E.3. Motion Forecasting

  • Motion queries combine contextual features with positional priors derived from scene-level anchors transformed into each agent’s coordinate frame.Anchors are clustered from all agents’ training data using k-means with K = 6; the agent’s current position and the previous layer’s predicted goal also contribute to query position. MotionFormer uses D = 256, predicts trajectories over T = 12 steps (6 seconds), and stacks N = 3 modules.Source: §E.3 · p. 17
  • MotionFormer uses agent-agent, agent-map, and agent-goal interaction modules to model agent interactions, map context, and trajectory goals.The first two modules use standard transformer decoder layers; agent-goal interaction uses deformable cross-attention with a previously predicted trajectory goal as reference. It samples four points per trajectory and uses six trajectories per agent.Source: §E.3 · p. 17

E.4. Occupancy Prediction

  • OccFormer unrolls occupancy across five blocks, each predicting one frame in a temporal horizon that includes the current frame.The BEV feature is downsampled by 4 before entering OccFormer for efficient multi-step prediction.Source: §E.4 · p. 17
  • Occupancy prediction combines dense scene features with sparse agent features through pixel-level self-attention and masked scene-agent cross-attention.Attended dense features are upsampled and added residually; a convolutional decoder predicts occupancy at the original BEV resolution, while matrix multiplication of occupancy and decoded dense features generates instance-level occupancy. The mask-feature MLP, occupancy-feature MLP, and decoder are shared across blocks; other components are independent.Source: §E.4 · p. 18

E.5. Planning

UniAD’s planner combines tracking and motion-forecasting queries with a command embedding, interacts with dense BEV features, and regresses an ego trajectory. A collision optimizer uses predicted occupancy and trajectory for obstacle avoidance.

E.5. Planning

  • The planner aggregates tracking and motion-forecasting queries with the command embedding, then uses three transformer decoder layers to cross-attend dense BEV features.The queries are encoded with MLP layers and max-pooled across modalities; learned position embeddings are added to the plan query and sinusoidal embeddings to the BEV feature.Source: §E.5 · p. 18
  • The planner regresses a 6-step trajectory covering 3 seconds from the BEV-interacted plan query.The trajectory is denoted ˆτ ∈RTp×2, with Tp = 6.Source: §E.5 · p. 18
  • A collision optimizer uses predicted occupancy and trajectory for obstacle avoidance.Its settings are d = 5, σ = 1.0, λcoord = 1.0, and λobs = 5.0.Source: §E.5 · p. 18

E.6. Training Details

UniAD is trained in two stages, moving from perception pretraining to joint optimization of tracking, mapping, motion forecasting, occupancy prediction, and planning. Task-specific losses include a collision penalty intended to keep planned trajectories away from obstacles.

E.6. Training Details

  • UniAD uses two-stage training: 6 epochs of tracking and mapping after loading BEVFormer [55] weights, then 20 epochs of all five task losses with the image backbone and BEV encoder frozen.For ablations, stage two uses 8 epochs; tracking and BEV feature aggregation sequences are length 5 in both stages and length 3 in ablations.Source: §E.6 · pp. 18–19
  • Tracking uses a Hungarian loss combining focal classification and L1 box localization, with newborn queries matched to ground truth and track queries inheriting prior-frame assignments.The weights are λfocal = 2 and λl1 = 0.25.Source: §E.6 · p. 19
  • Online mapping combines focal classification, L1 box, GIoU, and Dice losses for lanes, dividers, contours, and drivable-area segmentation.The weights are λfocal = λgiou = λdice = 2 and λl1 = 0.25.Source: §E.6 · p. 19
  • Motion forecasting uses a multimodal Gaussian-mixture loss combining classification and negative log-likelihood terms, while predicting and accumulating agent speeds for temporal smoothness.Both loss weights are 0.5.Source: §E.6 · p. 19
  • Occupancy prediction uses binary cross-entropy and Dice losses for agent segmentation, with an auxiliary loss supervising the pixel-agent interaction attention mask.The respective weights are 5 and 1.Source: §E.6 · p. 19
  • Planning combines naive imitation L2 loss with a collision loss that penalizes overlap between the ego trajectory’s expanded bounding box and forecast agents.The weights are λimi = 1 and λcol = 2.5; safety-distance pairs are (1.0, 0.0), (0.4, 0.5), and (0.1, 1.0).Source: §E.6 · p. 19

F. Experiments · F.1. Protocols

UniAD experiments follow BEVFormer’s basic two-stage training settings, use a 200×200 BEV grid over a 102.4m square, and run on 16 NVIDIA Tesla A100 GPUs.

F.1. Protocols

  • Both training stages follow most BEVFormer [55] settings, with batch size 1, learning rate 2×10−4, backbone learning-rate multiplier 0.1, and AdamW [65] with 1×10−2 weight decay.Experiments use 16 NVIDIA Tesla A100 GPUs.Source: §F.1 · pp. 19–20
  • The default BEV grid is 200×200, covering [-51.2m, 51.2m] on each of the X and Y axes at 0.512m intervals.Additional feature-dimension hyperparameters are reported in Table 11.Source: §F.1 · p. 20

F.2. Metrics · F.3. Model complexity and Computational cost

The section specifies evaluation metrics and conditions for tracking, mapping, motion forecasting, occupancy prediction, and planning. It also reports that UniAD’s computational complexity is mainly in the encoder and that it has lower FLOPs than BEVerse despite supporting more tasks.

F.2. Metrics

  • Tracking uses AMOTA, AMOTP, Recall, and IDS; AMOTA integrates MOTA over recall, while AMOTP averages matched-track position error across recall.AMOTP uses the x- and y-axis position error of matched tracks and the true positives at each recall.Source: §F.2 · p. 20
  • Online mapping evaluates IoU separately for lanes, boundaries, pedestrian crossings, and drivable area by comparing network outputs with ground-truth maps.Source: §F.2 · p. 20
  • Motion forecasting reports minADE, minFDE, and MR on matched true positives at a 1.0m matching threshold, with a 2.0m miss-FDE threshold [57] [75].The motion-forecasting metrics are evaluated on the vehicle category.Source: §F.2 · p. 20
  • End-to-end forecasting also uses EPA with ViP3D’s setting [30] and minFDE-AP [75], counting a true positive only within 1.0m perception and 2.0m min-FDE thresholds.For minFDE-AP, ground truth is not split into static, linear, and nonlinear motion bins; vehicle classes are merged.Source: §F.2 · p. 20
  • Occupancy prediction uses instance-agnostic IoU and instance-aware VPQ [35] [105] [48], evaluating current and four future steps across near 30m×30m and far 100m×100m BEV ranges.The four-step future horizon covers 2.0 seconds at 2Hz, including the current timestamp.Source: §F.2 · p. 20
  • Planning is evaluated using L2 error and collision rate at various timestamps, following ST-P3 [38].Source: §F.2 · p. 20

F.3. Model complexity and Computational cost

  • UniAD supports more tasks, achieves superior performance, and has lower FLOPs than BEVerse, while computational complexity mainly comes from the encoder rather than task decoders.The comparison is reported alongside a complexity and runtime measurement on an Nvidia Tesla A100 GPU.Source: §F.3 · p. 20

F.4. Model scale

UniAD is evaluated at three model scales, with larger image encoders associated with improved perception and downstream prediction and planning. Its reported failure cases occur mainly in long-tail scenarios that affect all modules.

F.4. Model scale

  • UniAD failure cases occur mainly in long-tail scenarios, where all modules are affected.The paper points to Fig. 13 and Fig. 14 for these cases.Source: §F.4 · p. 21
  • ResNet-50, ResNet-101, and VoVNet 2-99 serve as the image encoders for UniAD-S, UniAD-B, and UniAD-L, respectively, and larger backbones improve perceptual scores and downstream prediction and planning.The paper presents three UniAD scale variations in Table 12.Source: §F.4 · pp. 20–21, Table 12 · p. 21

F.5. Qualitative results

Qualitative visualizations show command-dependent planning attention and trajectory changes, safe and attentive behavior across driving scenarios, recovery despite perception misses, and remaining long-tail failures.

Attention mask visualization

  • Different navigation commands produce substantially different planned trajectories, while attention concentrates on the goal lane and critical agents yielding to the ego vehicle.The visualization shows two consecutive frames with different commands and overlays the planner’s attention mask, tracking boxes, and planned trajectory.Source: §F.5 · p. 21, Figure 8 · p. 22

Visualization of different scenarios

  • In urban cruising, UniAD produces interpretable perception and prediction results and makes a safe maneuver.The visualization includes six camera views, predicted results, and the planning module’s attention mask.Source: Figure 9 · p. 22
  • In critical cases, the ego vehicle yields to crossing pedestrians and a fast-moving car, with attention on those agents and the intended goal location.The second case involves waiting to go straight without protection near an intersection.Source: Figure 10 · p. 23
  • To avoid obstacle vehicles, the ego vehicle changes lanes attentively, while planner attention focuses on the obstacles and road ahead and behind.The figure shows two obstacle-avoidance scenarios.Source: Figure 11 · p. 23
  • The planner attends to critical agents missed by tracking and reacts to them, illustrating recovery from prior-module errors.A fast-moving vehicle is initially missed and later tracked; a stationary car is missed in both frames, yet both elicit strong reactions in the planner’s attention masks.Source: §F.5 · p. 21, Figure 12 · p. 24

Failure cases

  • Long-tail failures include inaccurate trailer size and vehicle heading estimates, and over-cautious planning around an oncoming vehicle in a dark narrow street.The paper suggests that a smaller collision-loss weight and more boundary regulation might mitigate the over-cautious behavior.Source: Figure 13 · p. 24, Figure 14 · p. 24

Related papers

Loading 2212.10156v2…