PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing End-to-End Autonomous Driving with Latent World Model

Yingyan Li, Lue Fan, Jiawei He, Yuqi Wang, Yuntao Chen, Zhaoxiang Zhang, Tieniu Tan

arXiv:2406.08481v2cs.CV

TL;DR

End-to-end driving needs richer scene representations from raw sensor data. The paper introduces LAW, which predicts future scene features from current features and ego trajectories within perception-free and perception-based frameworks. LAW achieves state-of-the-art performance across nuScenes, NAVSIM, and CARLA.

  • Problem

    End-to-end planners reduce information loss by using raw sensor data, motivating better scene feature representations for fully leveraging those data.

  • Method

    LAW is a self-supervised latent world model that predicts future scene latents from current scene latents and ego trajectories.

  • Results

    LAW achieves state-of-the-art performance on nuScenes, NAVSIM, and CARLA and enhances both perception-free and perception-based frameworks.

  • Takeaways & Limitations

    LAW provides a unified self-supervised approach that jointly enhances scene representation learning and ego trajectory prediction across two end-to-end driving paradigms.

Abstract

from arXiv · show

In autonomous driving, end-to-end planners directly utilize raw sensor data, enabling them to extract richer scene features and reduce information loss compared to traditional planners. This raises a crucial research question: how can we develop better scene feature representations to fully leverage sensor data in end-to-end driving? Self-supervised learning methods show great success in learning rich feature representations in NLP and computer vision. Inspired by this, we propose a novel self-supervised learning approach using the LAtent World model (LAW) for end-to-end driving. LAW predicts future scene features based on current features and ego trajectories. This self-supervised task can be seamlessly integrated into perception-free and perception-based frameworks, improving scene feature learning and optimizing trajectory prediction. LAW achieves state-of-the-art performance across multiple benchmarks, including real-world open-loop benchmark nuScenes, NAVSIM, and simulator-based closed-loop benchmark CARLA. The code is released at https://github.com/BraveGroup/LAW.

1 INTRODUCTION

End-to-end planners use raw sensor data to reduce information loss, but richer scene representations remain needed. LAW addresses this with self-supervised future latent prediction and supports both major framework types while achieving state-of-the-art benchmark performance.

  • End-to-end planners directly use raw sensor data, reducing information loss relative to traditional planners that consume pre-processed perception outputs.
  • Self-supervised learning uses temporal information to enrich scene feature learning for continuous-video autonomous driving.
  • LAW predicts future scene latents from current scene latents and ego trajectories, using future-frame features as training supervision.
  • The self-supervised task jointly enhances scene representation learning and ego trajectory prediction.
  • LAW supports perception-free perspective-view features and perception-based BEV features across common autonomous-driving frameworks.
  • LAW achieves state-of-the-art performance on nuScenes, NAVSIM, and CARLA benchmarks.

2 RELATED WORKS

Related work covers perception-based and perception-free end-to-end methods, alongside image-based and occupancy-based autonomous-driving world models. LAW differs from these world models by requiring no manual annotations.

  • End-to-end autonomous-driving methods are divided into perception-based and perception-free categories according to whether they perform perception tasks.
  • Perception-free methods avoid large numbers of perception annotations and have included reinforcement-learning-based approaches.
  • Autonomous-driving world models are categorized as image-based or occupancy-based models.
  • Image-based world models generate future driving imagery, whereas occupancy-based models predict occupancy and require occupancy annotations.
  • LAW predicts latent features without manual annotations, distinguishing it from occupancy-based world models.

3 PRELIMINARY

The preliminary formulation represents driving observations with multi-view images and predicts future ego waypoints. A world model then predicts the next state from the current state and planned waypoints.

  • Vision-based End-to-end Autonomous Driving: The driving input is a set of N surrounding multi-view images captured at time step t.
  • Vision-based End-to-end Autonomous Driving: The planner predicts a sequence of M future ego waypoints representing BEV positions at later time steps.
  • World Model: A world model predicts future states from the current state and the ego action sequence.
  • World Model: In this formulation, the world model predicts state S_t+1 using current state S_t and predicted waypoints W_t.

4 METHODOLOGY

LAW adds a self-supervised future-latent prediction task to end-to-end driving by combining current visual latents with predicted waypoints. The design supports both perception-free perspective-view and perception-based BEV frameworks, with framework-specific decoders and supervision.

  • Latent World Model: LAW extracts current visual latents from images, predicts waypoints, and uses both to predict visual latents for the future frame.Future visual latents provide the training target for the latent world model.
  • Latent World Model: Action-aware latents are formed by concatenating each visual latent with a waypoint vector before latent-world-model prediction.The concatenated representation is processed by an MLP and retains the visual latent shape.
  • Latent World Model: A transformer-based latent world model predicts future visual latents from the action-aware latents using self-attention across latent feature vectors.The prediction is trained against future visual latents with mean squared error supervision.
  • Framework Compatibility: The latent world model is compatible with both perception-free and perception-based end-to-end driving frameworks.This compatibility enables the same self-supervised future-latent prediction principle across perspective-view and BEV representations.
  • Perception-Free Framework: The perception-free framework produces perspective-view latents from multi-view images and decodes them into waypoints through learnable queries and cross-attention.Its supervision relies solely on ground-truth waypoints and uses an L1 waypoint loss.
  • Perception-Based Framework: The perception-based framework flattens BEV feature maps and predicts waypoints using motion-prediction and map-construction tasks.Its final training objective combines waypoint supervision with motion-prediction and map-construction losses.

5 EXPERIMENTS

LAW is evaluated across nuScenes, NAVSIM, and CARLA using perception-free and perception-based frameworks, with benchmark comparisons and ablations examining latent prediction, time horizons, and architecture. The experiments report state-of-the-art or improved performance across these settings and show that action-conditioned latent prediction and feature interactions matter.

  • Benchmarks: LAW is evaluated on nuScenes, NAVSIM, and CARLA, covering real-world open-loop and simulator-based closed-loop driving.nuScenes and NAVSIM provide open-loop evaluation, while CARLA provides closed-loop evaluation with continuously updated sensor inputs.
  • Experimental settings: The experiments implement both perception-free and perception-based frameworks, with LAW supporting perspective-view or BEV feature prediction respectively.The reported benchmark settings include perception-free implementations for nuScenes, NAVSIM, and CARLA, and a perception-based nuScenes comparison.
  • Benchmark results: LAW achieves state-of-the-art results on nuScenes, NAVSIM, and CARLA, while its perception-free CARLA approach surpasses methods using extensive auxiliary supervision.On nuScenes, the perception-based framework is state of the art in L2 displacement and collision rates; NAVSIM achieves state-of-the-art PDMS; CARLA outperforms existing methods.
  • Ablation studies: Ablations show that accurate future latent prediction depends on driving actions, while latent prediction improves NAVSIM PDMS and CARLA Driving Score.NAVSIM gains are mainly associated with drivable area compliance and ego progress improvements.
  • Design analysis: The best latent-prediction horizon is 1.5 seconds, and feature interactions between different positions are important for the default network architecture.The tested horizons include 0.5, 1.5, 3.0, and 10.0 seconds; a single-layer projection performs poorly, while the default architecture performs best among tested architectures.
  • Visualization: A visualization shows LAW capturing scene information that VAD overlooks, enabling a safer predicted forward trajectory in the illustrated case.The figure highlights a rear-end collision for VAD using yellow circles.

6 CONCLUSION

The paper presents a latent world model that predicts future features from current features and ego trajectories for end-to-end autonomous driving. It supports both perception-free and perception-based frameworks and achieves state-of-the-art results on nuScenes, NAVSIM, and CARLA.

  • The latent world model predicts future features from current features and ego trajectories as a self-supervised method for end-to-end autonomous driving.
  • The method accommodates both perception-free and perception-based frameworks by predicting perspective-view features and BEV features, respectively.
  • The approach achieves state-of-the-art results on the nuScenes, NAVSIM, and CARLA benchmarks.

A.1 PREDICTING MULTIPLE FEATURES USING MULTIPLE INPUT FRAMES

The appendix examines predicting multiple future frame latents with auto-regressive prediction and with multiple historical input frames. The experiments use temporal latent prediction settings, including a two-stage training procedure for multiple inputs.

  • Predicting multiple future features: The latent world model predicts multiple future frame latents auto-regressively, using each predicted latent to continue future prediction.The experiment uses only the front-view camera to facilitate fast training.
  • Predicting multiple future features: Table 8 reports results for predicting multiple future latents in an auto-regressive manner.
  • Predicting multiple future features using multiple input frames: Multiple input frame latents are incorporated to leverage temporal information more effectively when predicting multiple future frame latents.
  • Predicting multiple future features using multiple input frames: The multiple-input experiment adopts a two-stage training paradigm, beginning with single input frame latents for 12 epochs to improve convergence.
  • Predicting multiple future features using multiple input frames: Table 9 reports results for predicting future latents with multiple history frame inputs.

A.2 MORE VISUALIZATION

The appendix provides additional visualizations comparing the paper’s map construction and agent motion prediction with VAD. The comparisons particularly address heavily occluded and crowded conditions.

  • More visualization: The supplementary materials include a demo based on the CARLA simulator.
  • More visualization: Figure 4 compares the paper’s map construction results with VAD and reports visibly better results for the paper’s method.
  • More visualization: Figure 5 compares the paper’s agent motion prediction results with VAD and reports visibly better results for the paper’s method.
  • More visualization: Figure 6 compares both map construction and agent motion prediction with VAD, especially under heavily occluded and crowded conditions.
Loading 2406.08481v2…