PaperlayerView exact revision on arXiv

Source-linked AI summary

Planning-oriented Autonomous Driving

Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, Hongyang Li

arXiv:2212.10156v2cs.CVcs.RO

TL;DR

Autonomous-driving systems can suffer from error accumulation, negative transfer, or incomplete coordination when tasks are isolated or only loosely integrated. UniAD organizes perception and prediction around planning with query-based task interfaces, and its experiments report superior performance across essential driving metrics, while comprehensive coordination remains computationally demanding.

  • Problem

    Isolated and multi-task autonomous-driving designs can suffer from information loss, error accumulation, feature misalignment, or negative transfer, motivating better coordination toward planning.

  • Method

    UniAD is a planning-oriented end-to-end framework that connects perception, prediction, and planning through query-based interfaces and coordinated task representations.

  • Results

    UniAD outperforms prior methods across essential perception, prediction, and planning metrics, including reductions of 51.2% in planning L2 error and 56.3% in collision rate versus ST-P3.

  • Takeaways & Limitations

    Planning-oriented task coordination provides a supported framework for jointly organizing diverse autonomous-driving tasks toward planning.

  • Takeaways & Limitations

    Coordinating multiple tasks, especially with temporal history, requires extensive computational power, and lightweight deployment remains future work.

Abstract

from arXiv · show

Modern autonomous driving system is characterized as modular tasks in sequential order, i.e., perception, prediction, and planning. In order to perform a wide diversity of tasks and achieve advanced-level intelligence, contemporary approaches either deploy standalone models for individual tasks, or design a multi-task paradigm with separate heads. However, they might suffer from accumulative errors or deficient task coordination. Instead, we argue that a favorable framework should be devised and optimized in pursuit of the ultimate goal, i.e., planning of the self-driving car. Oriented at this, we revisit the key components within perception and prediction, and prioritize the tasks such that all these tasks contribute to planning. We introduce Unified Autonomous Driving (UniAD), a comprehensive framework up-to-date that incorporates full-stack driving tasks in one network. It is exquisitely devised to leverage advantages of each module, and provide complementary feature abstractions for agent interaction from a global perspective. Tasks are communicated with unified query interfaces to facilitate each other toward planning. We instantiate UniAD on the challenging nuScenes benchmark. With extensive ablations, the effectiveness of using such a philosophy is proven by substantially outperforming previous state-of-the-arts in all aspects. Code and models are public.

1. Introduction

Autonomous-driving systems have evolved from isolated task models and shared-backbone multi-task designs toward end-to-end frameworks, but existing approaches face information loss, error accumulation, negative transfer, or incomplete task coordination. UniAD proposes a planning-oriented framework that organizes perception and prediction around planning and connects tasks through query-based interfaces.

  • Standalone models simplify development but risk information loss across modules, error accumulation, and feature misalignment from isolated optimization targets.
  • Multi-task learning shares a feature extractor across task-specific heads, enabling feature abstraction, task expansion, and lower onboard computation cost, but may cause negative transfer.
  • End-to-end autonomous driving unites perception, prediction, and planning, with preceding-task choice and priority determined in favor of planning.
  • Direct trajectory prediction without explicit perception or prediction supervision has shown promise in closed-loop simulation but is inadequate for safety guarantee and interpretability in highly dynamic urban scenarios.
  • UniAD introduces a comprehensive planning-oriented framework whose query interfaces connect tasks, support flexible interaction modeling, and coordinate perception, prediction, and planning.
  • UniAD is instantiated on a challenging benchmark and evaluated through extensive ablations against prior state-of-the-art methods.

2. Methodology

UniAD organizes perception, prediction, and planning as a connected, planning-oriented pipeline built around transformer decoder modules and task queries. Its modules exchange agent, map, motion, occupancy, and ego-vehicle information to produce future trajectories and planning outputs.

  • Framework overview: UniAD uses four transformer decoder-based perception and prediction modules followed by a planner, with task queries connecting pipeline nodes.Multi-camera images become unified BEV features before downstream query-based processing.
  • Perception modules: TrackFormer uses detection and track queries to jointly detect and track agents, while an ego-vehicle query explicitly represents the self-driving vehicle for planning.The ego-vehicle query is passed onward for interaction with other agents.
  • Perception modules: MapFormer represents lanes, dividers, crossings, and drivable area with map queries whose location and structure knowledge supports downstream motion forecasting.Its stacked layers are supervised, with updated map queries forwarded to MotionFormer.
  • Motion forecasting: MotionFormer jointly predicts top-k multimodal trajectories for all agents in a scene while modeling agent-agent, agent-map, and agent-goal interactions.It also passes the ego-vehicle query through the motion module so the ego vehicle interacts with other agents.
  • Motion forecasting: Motion queries combine preceding query context with scene-level anchors, agent-level anchors, the agent’s current location, and its predicted goal point.Anchors are clustered from ground-truth trajectory endpoints, while predicted endpoints are refined layer by layer.
  • Occupancy prediction: OccFormer predicts future instance-wise occupancy by combining dense scene features with agent-level features through attention and matrix multiplication.Its sequential blocks unroll occupancy across future horizons using motion-query information and prior dense features.

3. Experiments

On nuScenes, UniAD is evaluated as a jointly trained full-stack system, with ablations testing planning-oriented task coordination. It outperforms prior methods across perception, prediction, and planning while revealing specific module contributions and long-tail limitations.

  • Joint results: Both motion and occupancy prediction are required for the best planning L2 and collision-rate results compared with naive end-to-end planning.Their joint inclusion also improves both agent-level and scene-level prediction metrics.
  • Joint results: UniAD outperforms naive multi-task learning across motion, occupancy, and planning metrics, including -15.2% minADE, -17.0% minFDE, -0.15m avg.L2, and -0.51 avg.Col.(%).The ablation supports the superiority of the planning-oriented design over separate task heads.
  • Experimental setup: UniAD jointly performs perception, prediction, and planning with a single trained network on the nuScenes validation set.Experiments compare joint results, modular results against prior methods, and design ablations.
  • Modular results: +6.5 and +14.2 AMOTA(%) improvements over MUTR3D and ViP3D, respectively, are reported for multi-object tracking, while online lane segmentation improves by +7.4 IoU(%) over BEVFormer.Tracking also achieves the lowest ID switch score, although some perception-oriented methods remain stronger on specific mapping classes.
  • Modular results: 38.3% and 65.4% reductions in minADE versus PnPNet-vision and ViP3D accompany occupancy gains of +4.0 and +2.0 IoU-near(%) versus FIERY and BEVerse.The occupancy improvements are concentrated in nearby areas, which are more critical for planning.
  • Planning and ablations: 51.2% lower planning L2 error and 56.3% lower collision rate than ST-P3 are reported, while ablations show contributions from scene-centric anchors, goal interaction, uncertainty-aware optimization, attention masks, and mask-feature reuse.Qualitative analysis indicates later tasks can sometimes recover from inaccurate upstream predictions, whereas failures mainly occur in long-tail scenarios such as large trucks and trailers.

4. Conclusion and Future Work

UniAD presents a planning-oriented autonomous-driving framework that coordinates perception, prediction, and planning through a unified query-based design. The authors identify computational cost and future task integration as remaining challenges.

  • 4. Conclusion and Future Work: UniAD organizes multiple driving tasks around planning and connects them through query-based interfaces that provide richer representations for agent interaction.The framework is presented as a system-level design rather than a simple stack of independently engineered tasks.
  • 4. Conclusion and Future Work: Extensive experiments verify the proposed method across the evaluated aspects.
  • 4. Conclusion and Future Work: Comprehensive multi-task coordination requires substantial computational power, particularly when temporal history is used.
  • 4. Conclusion and Future Work: Future work includes lightweight deployment and integrating additional tasks such as depth estimation and behavior prediction.

A. Task Definition

UniAD defines a full-stack driving pipeline in which perception produces structured scene representations, prediction models future dynamics, and planning uses upstream information to generate the ego trajectory.

  • A. Task Definition: Detection locates surrounding 3D objects, while tracking associates them across time using consistent track IDs and forwards associated boxes and features to motion forecasting.
  • A. Task Definition: Online mapping segments lanes, drivable areas, dividers, and pedestrian crossings in bird’s-eye view from onboard multi-view images.
  • A. Task Definition: Motion forecasting predicts agents’ future trajectories and bridges perception with planning to support final driving safety.
  • A. Task Definition: Occupancy prediction forecasts how a dense bird’s-eye-view grid changes over future timesteps, complementing sparse-agent motion forecasting with whole-scene representation.
  • A. Task Definition: Planning consumes upstream results, including the ego-vehicle query and dense bird’s-eye-view features, to predict an ego trajectory.

B. The Necessity of Each Task

The paper motivates coordinating perception and prediction tasks for planning because complementary agent-level and scene-level representations can support safer, more informed decisions. It contrasts this design with modular independence and direct trajectory prediction.

  • B. The Necessity of Each Task: Tracking in the loop supplies spatial-temporal features and history tracks for occluded agents, while maps and motion forecasting support more accurate planning.
  • B. The Necessity of Each Task: UniAD combines agent-centric motion forecasting with whole-scene occupancy because dense features and sparse agent features provide complementary prediction information.
  • B. The Necessity of Each Task: Joint perception-prediction learning has been proposed to avoid cascading errors in traditionally modular pipelines.
  • B. The Necessity of Each Task: Occupancy-grid methods represent future scene states densely, but some prior approaches cannot capture multimodal behaviors or rely on deterministic future segmentation.
  • B. The Necessity of Each Task: End-to-end planning work has explored direct control or trajectory prediction, whereas UniAD incorporates tracking and two prediction representations in one end-to-end model.

E. Implementation Details

The implementation builds UniAD from query-based detection, tracking, mapping, motion forecasting, occupancy prediction, and planning components. MotionFormer uses multiple interaction blocks, while OccFormer generates sequential occupancy representations from dense and agent-level features.

  • E. Implementation Details: Detection uses BEVFormer-style BEV features and a Deformable DETR head, while track queries maintain previously observed instances for end-to-end tracking.
  • E. Implementation Details: During training, detection queries are Hungarian-matched and updated through a query interaction module to become track queries at subsequent timestamps.
  • E. Implementation Details: During inference, frames are processed sequentially and classification scores filter detection and track queries instead of ground-truth-dependent 3D IoU.
  • E. Implementation Details: Online mapping separates instance-wise thing queries from a class-fixed stuff query, using 300 thing queries and 1 stuff query for drivable area.
  • E. Implementation Details: MotionFormer transforms scene-level anchors into agent coordinates, combines prior positions and query context, and refines trajectories layer by layer.
  • E. Implementation Details: MotionFormer stacks agent-agent, agent-map, and agent-goal interaction transformers, with deformable cross-attention sampling BEV features near predicted goal points.
  • E. Implementation Details: MotionFormer uses Gaussian mixture trajectories with K = 6 modalities and a prediction horizon T of 12 steps, corresponding to 6 seconds.
  • E. Implementation Details: OccFormer uses To = 5 sequential blocks to generate occupancy for current and future frames, combining dense scene features with sparse agent features.

E.5. Planning

The planner combines ego-vehicle queries from tracking and motion forecasting with a command embedding, then aggregates modality features before BEV interaction. A collision optimizer uses predicted occupancy and trajectories for obstacle avoidance.

  • Planner: Tracking and motion-forecasting ego queries are combined with a command embedding through MLPs and max-pooling across modalities.The max-pooling selects and aggregates the most salient modal features.
  • OccFormer: OccFormer generates occupancy across To sequential blocks, with each block responsible for one frame in the temporal horizon.Dense scene features and sparse agent features inject agent-level knowledge into future scene representations.
  • OccFormer: Instance-level occupancy is formed through matrix multiplication between agent-level features and decoded dense features at each block’s end.
  • Planner: The collision optimizer takes predicted occupancy and trajectory as inputs for obstacle avoidance.Its settings are d = 5, σ = 1.0, λcoord = 1.0, and λobs = 5.0.

E.6. Training Details

UniAD uses two-stage joint learning: perception is pretrained first, then downstream tasks are trained jointly while selected image and BEV components remain frozen. The objective combines tracking, mapping, motion, occupancy, and planning losses.

  • Joint learning: Stage one pretrains tracking and online mapping, loading BEVFormer weights while stopping gradient back-propagation in the image backbone.The loaded components include the image backbone, FPN, BEV encoder, and detection decoder, excluding object query embeddings.
  • Joint learning: The first-stage objective is L_1 = L_track + L_map.
  • Joint learning: Stage two freezes the image backbone and BEV encoder, then trains tracking, mapping, motion forecasting, occupancy prediction, and planning jointly.Joint training lasts 20 epochs, while ablation studies use 8 epochs for efficiency.
  • Task losses: Tracking uses a Hungarian loss combining Focal classification and l1 3D-box localization losses.Newborn queries are bipartite-matched to ground-truth objects, while track queries inherit assignments from previous frames.
  • Task losses: Mapping combines Focal, L1, GIoU, and Dice losses for lanes, dividers, contours, and drivable area.
  • Task losses: Motion forecasting models multimodal trajectories as Gaussian mixtures with classification and negative log-likelihood terms.Agent speeds are predicted per timestep and accumulated to obtain final trajectories.
  • Task losses: Occupancy prediction uses binary cross-entropy and Dice losses, with an auxiliary loss supervising the pixel-agent interaction attention mask.
  • Task losses: Planning combines imitation L2 loss with collision loss to keep planned trajectories away from obstacles.The collision optimizer uses predicted occupancy and trajectories with enlarged ego boxes for additional safety distance.

F. Experiments

Experiments follow BEVFormer training settings and use a 200×200 BEV representation over a 102.4m × 102.4m square. Training runs on 16 NVIDIA Tesla A100 GPUs.

  • Implementation settings: The experiments use batch size 1, learning rate 2×10^-4, backbone learning-rate multiplier 0.1, AdamW, and weight decay 1×10^-2.
  • Implementation settings: The default BEV grid is 200×200 with 0.512m intervals covering [-51.2m, 51.2m] on both X and Y axes.
  • Implementation settings: Experiments are conducted with 16 NVIDIA Tesla A100 GPUs.

F.2. Metrics

The experiments evaluate tracking, mapping, motion forecasting, occupancy prediction, planning, and system complexity with task-specific metrics. Occupancy is assessed across current and future frames at near and far BEV ranges, while the system-level comparison reports broader task coverage and lower FLOPs.

  • Multi-object tracking: Tracking is evaluated with AMOTA, AMOTP, Recall, and IDS on the nuScenes dataset.AMOTA and AMOTP integrate MOTA and MOTP over recall levels.
  • Multi-object tracking: AMOTA averages MOTA across recall levels, with MOTA accounting for false positives, false negatives, and identity switches.GT denotes the number of ground-truth objects in the frame.
  • Multi-object tracking: AMOTP averages matched-track position errors over recall levels.
  • Online mapping: Online mapping reports IoU for lanes, boundaries, pedestrian crossings, and drivable area.
  • Motion forecasting: Motion forecasting uses minADE, minFDE, MR, EPA, and minFDE-AP, with matching thresholds of 1.0m and a miss FDE threshold of 2.0m.The end-to-end metrics complement conventional displacement and miss-rate measures.
  • Occupancy prediction: Occupancy quality is measured by whole-scene IoU and instance-aware VPQ, which captures instance presence and temporal consistency.
  • Occupancy prediction: Occupancy metrics cover 2.0s at 2Hz using the current frame and four future steps across near 30m×30m and far 100m×100m BEV ranges.The temporal horizon is H = 4, yielding To = 5 timestamps including the current timestamp.
  • Planning: Planning is evaluated with L2 error and collision rate at various timestamps.

F.4. Model scale

Model scale is examined through three UniAD variants, with larger image backbones improving perceptual quality and subsequently prediction and planning performance. The qualitative failures remain concentrated in long-tail scenarios affecting all modules.

  • Model scale: Three UniAD variations are compared at different model scales, with Table 12 reporting their model configurations.The comparison includes UniAD-S, UniAD-B, and UniAD-L.
  • Failure cases: UniAD’s principal failure cases occur in long-tail scenarios where all modules are affected.The paper presents these cases to expose weaknesses and guide future work.
  • Model scale: Table 13 compares computational complexity and runtime as different modules are incorporated.ID.1 is similar to original BEVFormer, while ID.0 is an MTL framework.
  • Model scale: UniAD-S, UniAD-B, and UniAD-L use ResNet-50, ResNet-101, and VoVNet 2-99 image backbones, respectively.
  • Model scale: Larger image backbones improve perceptual scores and can further improve prediction and planning performance through better BEV feature quality.

F.5. Qualitative results

Qualitative visualizations show UniAD adapting attention and planned trajectories to navigation commands, goals, critical agents, and obstacles. They also illustrate interpretable urban maneuvers, recovery after an upstream tracking miss, and long-tail failure modes.

  • Attention and navigation: Planned trajectories vary with navigation commands, while planner attention concentrates on the goal lane and critical agents yielding to the ego vehicle.
  • Urban cruising: Urban-cruising visualizations combine six camera views with predicted results and planner attention masks to illustrate interpretable perceptual, predictive, and planning outputs.Top-1 and top-3 motion-forecasting trajectories are shown in image-view and BEV visualizations, respectively.
  • Critical cases: In critical cases, attention focuses on pedestrians, fast-moving vehicles, and the intended goal location while the ego vehicle yields.
  • Obstacle avoidance: During lane changes, attention focuses on obstacle vehicles and the road ahead and behind.
  • Recovery and failures: A later task can recover from an earlier perception failure when tracking initially misses a fast-moving vehicle and captures it in the subsequent frame.The visualization uses two consecutive frames from the same scenario.
  • Recovery and failures: Long-tail failures include inaccurate trailer size and vehicle headings, plus over-cautious planning around an incoming vehicle in a dark narrow street.The paper suggests smaller collision-loss weighting and stronger boundary regulation as possible mitigations for the over-cautious planning case.
Loading 2212.10156v2…