PaperlayerView exact revision on arXiv

Source-linked AI summary

DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Driving

Xiaosong Jia, Junqi You, Zhiyuan Zhang, Junchi Yan

arXiv:2503.07656v2cs.LGcs.CVcs.RO

TL;DR

Existing E2E-AD systems often rely on sequential task pipelines and dense BEV representations, creating training and computational challenges. DriveTransformer addresses these issues with parallel task queries, direct raw-sensor interaction, and streaming history, and reports state-of-the-art performance on Bench2Drive and nuScenes with high FPS. Its unified task updates improve integration but leave the system coupled and harder to maintain or debug.

  • Problem

    Sequential perception-prediction-planning pipelines and dense BEV representations limit task synergy while creating training instability and computational challenges for long-range perception and temporal fusion.

  • Method

    DriveTransformer uses task parallelism, sparse task-query interaction with raw sensor features, and streaming FIFO history, implemented through task self-attention, sensor cross-attention, and temporal cross-attention.

  • Results

    DriveTransformer achieves state-of-the-art performance on both closed-loop Bench2Drive and open-loop nuScenes evaluations with high FPS.

  • Takeaways & Limitations

    The reported results support a unified Transformer paradigm for efficient and scalable end-to-end autonomous driving.

  • Takeaways & Limitations

    DriveTransformer entangles updates across subtasks, making the overall system harder to maintain and debug separately.

Abstract

from arXiv · show

End-to-end autonomous driving (E2E-AD) has emerged as a trend in the field of autonomous driving, promising a data-driven, scalable approach to system design. However, existing E2E-AD methods usually adopt the sequential paradigm of perception-prediction-planning, which leads to cumulative errors and training instability. The manual ordering of tasks also limits the system`s ability to leverage synergies between tasks (for example, planning-aware perception and game-theoretic interactive prediction and planning). Moreover, the dense BEV representation adopted by existing methods brings computational challenges for long-range perception and long-term temporal fusion. To address these challenges, we present DriveTransformer, a simplified E2E-AD framework for the ease of scaling up, characterized by three key features: Task Parallelism (All agent, map, and planning queries direct interact with each other at each block), Sparse Representation (Task queries direct interact with raw sensor features), and Streaming Processing (Task queries are stored and passed as history information). As a result, the new framework is composed of three unified operations: task self-attention, sensor cross-attention, temporal cross-attention, which significantly reduces the complexity of system and leads to better training stability. DriveTransformer achieves state-of-the-art performance in both simulated closed-loop benchmark Bench2Drive and real world open-loop benchmark nuScenes with high FPS.

1 INTRODUCTION

DriveTransformer replaces sequential, BEV-heavy E2E-AD designs with a unified Transformer framework built around parallel task interaction, sparse sensor access, and streaming temporal fusion. It targets training stability, computational efficiency, and scalability while achieving state-of-the-art results on closed-loop Bench2Drive and open-loop nuScenes.

  • Approach: At each layer, task tokens use self-attention for cross-task interaction, sensor cross-attention for raw inputs, and temporal cross-attention for history fusion.This framework uses agent, map, and ego-related tokens initialized from learnable parameters, sensor processing, or canbus information.
  • Motivation: Existing E2E-AD methods commonly use sequential perception-prediction-planning pipelines, which can cause cumulative errors, training instability, and difficult multi-stage deployment.The sequential design also limits synergies between tasks and complicates industrial scaling.
  • Motivation: BEV-based methods face computational challenges for long-range detection and temporal fusion because dense grids and stored history BEV features are expensive.These methods also discard per-frame task queries and under-optimize image backbones because of weak gradient signals.
  • Approach: DriveTransformer introduces task parallelism, sparse representation, and streaming processing to unify cross-task, sensor, and temporal information exchange.Task queries interact with one another, raw sensor features, and FIFO history queues without an explicit task hierarchy.
  • Results: DriveTransformer achieves state-of-the-art closed-loop performance on Bench2Drive and state-of-the-art open-loop planning performance on nuScenes.The reported evaluations cover CARLA simulation and real-world nuScenes data.

2 RELATED WORKS

Related work spans direct image-to-control E2E-AD, unified task-based BEV systems, task-decoupled approaches, and sparse-query methods. DriveTransformer is positioned within this progression toward alternative task and representation designs.

  • End-to-End Autonomous Driving: Early E2E-AD methods mapped camera inputs directly to control commands, while later approaches added auxiliary supervision or privileged-input teacher training.Examples include CIL, CILRS, PlanT, and LBC.
  • Task-Based Methods: Explicit-subtask methods unified detection, prediction, and planning through BEV segmentation, Transformer-linked tasks, or vectorized representations.ST-P3, UniAD, and VAD represent these directions.
  • Alternative Designs: Other approaches removed task links or intermediate tasks, while concurrent work explored sparse query-based representations.ParaDrive removes links among tasks, BEVPlanner removes middle tasks, and several methods use sparse queries.

3 METHOD

DriveTransformer replaces sequential task dependencies with parallel query interactions, direct access to raw sensor tokens, and streaming history fusion. Its blocks unify sensor cross-attention, task self-attention, temporal cross-attention, and feed-forward processing while progressively refining task predictions.

  • Initialization & Tokenization: DriveTransformer represents detection, motion prediction, online mapping, and planning with task-specific queries that interact with one another, raw sensor inputs, and history.Agent, map, and ego queries correspond to dynamic objects, static scene elements, and ego-vehicle behavior.
  • Initialization & Tokenization: Sensor tokens are extracted from multi-view image features and augmented with 3D position encodings computed from sampled points along each camera ray.The encoding uses camera intrinsic and extrinsic matrices, pixel coordinates, and depth samples.
  • Token Interaction: Task self-attention removes manually imposed task dependencies, allowing interleaved relations among perception, prediction, mapping, and planning tasks to be learned through attention.The design is intended to support cross-task interactions such as planning-aware perception and interactive prediction and planning.
  • Token Interaction: Temporal processing stores prior task queries in per-task FIFO queues, retains top-K agent and map queries, and transforms historical positions into the current ego coordinate system.History queries serve as keys and values for temporal cross-attention; velocity-conditioned adaptive normalization provides motion compensation for agents.
  • Token Interaction: Each block applies sensor cross-attention, task self-attention, temporal cross-attention, and an FFN, with raw sensor tokens and history queues shared across blocks.Sensor cross-attention connects task queries to raw inputs, while task self-attention exchanges information across ego, agent, and map queries.
  • DETR-Style Task Head: Task heads progressively refine predictions and position encodings, while shared agent features support detection and motion prediction without explicit tracking.Planning uses mode embeddings for multimodal ego-trajectory prediction, and updated position encodings capture evolving spatial and semantic relations.

4 EXPERIMENTS

Experiments evaluate DriveTransformer on closed-loop Bench2Drive and open-loop nuScenes, comparing efficiency, scaling, architectural components, task heads, and robustness. The studies report competitive performance, lower latency, larger feasible batch sizes, and improved robustness under sensor perturbations.

  • Comparison with State-of-the-Art Methods: DriveTransformer achieves state-of-the-art closed-loop performance on Bench2Drive and state-of-the-art open-loop planning performance on nuScenes.The evaluations cover CARLA simulation and the real-world nuScenes dataset.
  • Comparison with State-of-the-Art Methods: DriveTransformer persistently outperforms state-of-the-art methods and has lower inference latency than UniAD and VAD.Its unified, sparse, and streaming design supports batch size 12 on an A800, versus batch sizes 1 for UniAD and 4 for VAD.
  • Scaling Study: Scaling the unified Transformer structure benefits planning most, while larger pretrained image backbones primarily improve perception.The scaling study varies decoder layers and hidden dimension alongside image-backbone scale.
  • Paradigm Design Study: All three attention types help, with Sensor Cross Attention especially important and Temporal Cross Attention having the least influence.Task Self-Attention improves driving score by allowing the ego query to use detected objects and map elements for planning.
  • Paradigm Design Study: Discarding auxiliary tasks causes performance decay, while one-stage training is sufficient and perception pretraining provides no advantage.The authors attribute the one-stage result to the absence of manual task dependencies.
  • Task Design and Training Studies: Removing middle-layer supervision collapses training, whereas local-coordinate prediction, point-level positional encoding, and multi-mode planning improve their respective task outcomes.Multi-mode planning particularly improves control in scenarios requiring subtle steering, while point-level positional encoding improves online mapping.
  • Robustness Analysis: DriveTransformer demonstrates significantly better robustness than VAD under camera crashes, calibration errors, motion blur, and Gaussian noise.The study evaluates robustness for both closed-loop planning and open-loop perception.

5 CONCLUSION

DriveTransformer is a unified Transformer paradigm for end-to-end autonomous driving built around task parallelism, streaming processing, and sparse representation. It achieves state-of-the-art performance on Bench2Drive and nuScenes with high FPS.

  • Conclusion: DriveTransformer combines task parallelism, streaming processing, and sparse representation in a unified Transformer-based E2E-AD paradigm.These designs are presented as the framework's defining features.
  • Conclusion: DriveTransformer achieves state-of-the-art performance on both Bench2Drive closed-loop evaluation and nuScenes open-loop evaluation with high FPS.The conclusion attributes the reported efficiency to the three unified design properties.

A IMPLEMENTATION DETAILS

The implementation trains DriveTransformer under a fixed Bench2Drive setup using AdamW and cosine annealing, with ResNet50 image backbones and different model variants for comparison and ablation.

  • Implementation Details: DriveTransformer models are trained for 30 epochs on 8 A800 GPUs using learning rate 1e-4, weight decay 0.05, dropout 0.1, AdamW, and cosine annealing.The Bench2Drive base set contains 1000 clips.
  • Implementation Details: DriveTransformer-Large is reported for state-of-the-art comparisons, while DriveTransformer-Base is used for ablations unless otherwise specified.The implementation uses ResNet50 image backbones.

B DEV10 BENCHMARK

Dev10 is a compact closed-loop benchmark derived from Bench2Drive to support faster development while preserving diverse driving behaviors, weathers, and towns. Its low variance is validated across three random seeds, while full 220-route evaluation remains the comparison protocol.

  • Benchmark Motivation: Bench2Drive contains 220 routes, and evaluating DriveTransformer-Large on 8 A800 GPUs can take 2–3 days.Dev10 was introduced to reduce the computational cost of iterative development.
  • Benchmark Construction: Dev10 selects one route for each of 10 high-level scenario types from Bench2Drive's 44 scenario types.The selection covers diverse weathers and towns, while grouping similar scenarios such as cut-in cases.
  • Benchmark Construction: The 10 high-level types cover parking exits, emergency braking, cut-ins, obstacle overtaking, yielding to emergency vehicles, reverse-lane overtaking, intersections, blocked intersections, sequential lane changes, and traffic merging.These categories represent distinct behaviors required in the listed scenarios.
  • Benchmark Validation: Across three seeds, Dev10 driving scores are 60.45, 59.20, and 58.99, while success rates are 0.3 for all seeds.The authors use these results to characterize Dev10 as having very low variance.
  • Evaluation Protocol: Full 220-route Bench2Drive evaluation is retained for method comparisons, while Dev10 is used for ablations to save computational resources.The authors also propose Dev10 as a validation set and plan to open-source its protocol.

C LIMITATIONS

DriveTransformer entangles updates across all subtasks, which creates challenges for maintaining the overall system. The authors identify reducing this coupling as an important future direction.

  • Entangled updates across subtasks make DriveTransformer challenging to maintain.

D COMPARISON OF MIDDLE TASKS

The paper compares DriveTransformer's middle-task performance on the nuScenes validation set in Table 10.

  • DriveTransformer's middle-task performance is evaluated on the nuScenes validation set.

E COMPARISON WITH CONCURRENT PARALLEL AND SPARSE BASED METHODS

The paper compares DriveTransformer with concurrent sparse-based methods and ParaDrive, focusing on planning quality and efficiency. The reported comparison indicates good L2 performance with high efficiency.

  • DriveTransformer is compared with SparseAD, SparseDrive, and ParaDrive as related efficient or sparse-based methods.
  • The comparison reports L2 (m), collision (%), and latency across 1-second, 2-second, and 3-second horizons.
  • DriveTransformer achieves good L2 with high efficiency.

F TRAINING STABILITY & MULTI-STAGE TRAINING

The paper contrasts DriveTransformer's attention-based task interaction with the multi-stage training used by several prior systems. DriveTransformer's direct access to sensor and temporal information is presented as supporting training stability, while prior one-stage alternatives can underfit.

  • UniAD uses a three-step strategy involving BEVFormer, TrackFormer and MapFormer, then joint training of all modules.
  • UniAD trained in one stage reaches an overall loss around 54 by about epoch 4, whereas official two-stage training reaches around 34.
  • One-stage UniAD training underfits all modules and produces a much higher final loss.
  • SparseAD and SparseDrive also divide training into multiple stages, with SparseDrive separating sparse perception learning from joint perception-planner training.
  • DriveTransformer learns task interactions through attention rather than manual ordering, allowing tasks to access sensor and temporal information directly during early training.
  • Pretraining perception provides no gain for DriveTransformer, which the paper presents as evidence of training stability.
Loading 2503.07656v2…