PaperlayerView exact revision on arXiv

Source-linked AI summary

Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Driving

Xiaosong Jia, Penghao Wu, Li Chen, Jiangwei Xie, Conghui He, Junchi Yan, Hongyang Li

arXiv:2305.06242v1cs.CV

TL;DR

End-to-end autonomous-driving systems often assign safety-critical region selection and future-situation inference to simple decoders despite heavily designed encoders. ThinkTwice predicts coarse behavior, retrieves behavior-conditioned salient features, imagines future scenes, and cascades refinement; it achieves state-of-the-art performance on two closed-loop CARLA benchmarks, while requiring substantial computation and GPU memory.

  • Problem

    Existing encoder-decoder systems leave safety-critical region selection and future-situation inference to the decoder, whose capacity is often much smaller than the encoder's.

  • Method

    ThinkTwice predicts coarse trajectory and action, retrieves encoder features around the predicted location, predicts future scene features conditioned on behavior, and refines the output in cascaded layers.

  • Results

    ThinkTwice achieves state-of-the-art performance on two competitive closed-loop autonomous-driving benchmarks in CARLA, with ablations demonstrating component effectiveness.

  • Takeaways & Limitations

    The study supports enlarging decoder capacity with dense supervision and spatial-temporal priors rather than relying only on larger conventional MLP or GRU decoders.

  • Takeaways & Limitations

    ThinkTwice has large MACs and GPU memory usage, which matters for deployment on edge devices with limited computational power and memory.

Abstract

from arXiv · show

End-to-end autonomous driving has made impressive progress in recent years. Existing methods usually adopt the decoupled encoder-decoder paradigm, where the encoder extracts hidden features from raw sensor data, and the decoder outputs the ego-vehicle's future trajectories or actions. Under such a paradigm, the encoder does not have access to the intended behavior of the ego agent, leaving the burden of finding out safety-critical regions from the massive receptive field and inferring about future situations to the decoder. Even worse, the decoder is usually composed of several simple multi-layer perceptrons (MLP) or GRUs while the encoder is delicately designed (e.g., a combination of heavy ResNets or Transformer). Such an imbalanced resource-task division hampers the learning process. In this work, we aim to alleviate the aforementioned problem by two principles: (1) fully utilizing the capacity of the encoder; (2) increasing the capacity of the decoder. Concretely, we first predict a coarse-grained future position and action based on the encoder features. Then, conditioned on the position and action, the future scene is imagined to check the ramification if we drive accordingly. We also retrieve the encoder features around the predicted coordinate to obtain fine-grained information about the safety-critical region. Finally, based on the predicted future and the retrieved salient feature, we refine the coarse-grained position and action by predicting its offset from ground-truth. The above refinement module could be stacked in a cascaded fashion, which extends the capacity of the decoder with spatial-temporal prior knowledge about the conditioned future. We conduct experiments on the CARLA simulator and achieve state-of-the-art performance in closed-loop benchmarks. Extensive ablation studies demonstrate the effectiveness of each proposed module.

1. Introduction

ThinkTwice addresses the imbalanced encoder-decoder design in end-to-end autonomous driving by enlarging decoder capacity and conditioning refinement on predicted behavior. Its cascaded decoder uses safety-critical feature retrieval, future-scene prediction, and dense supervision, achieving state-of-the-art results on two closed-loop CARLA benchmarks.

  • Motivation: Existing encoder-decoder systems leave safety-critical region selection and future-situation inference to relatively simple MLP or GRU decoders.The encoder processes broad sensor inputs without access to intended ego behavior, while the encoder is often substantially more elaborate.
  • Design principles: The method extends decoder capacity with spatial-temporal prior knowledge and dense supervision rather than simply increasing MLP depth or width.The proposed design aims to capture driving logical reasoning while avoiding the severe overfitting associated with simply enlarging MLPs.
  • Method: The cascaded decoder first predicts coarse future trajectory and action, retrieves features around the predicted location, and anticipates future scene representations conditioned on the prediction.The Look Module supplies goal-related scene features, while the Prediction Module generates future scene features.
  • Results: ThinkTwice achieves state-of-the-art performance on two competitive closed-loop autonomous driving benchmarks and evaluates its components through extensive ablation studies.The experiments use the CARLA simulator.
  • Contributions: ThinkTwice proposes a scalable decoder paradigm that emphasizes enlarging decoder capacity in end-to-end autonomous driving.The authors describe this as the first work in the field to emphasize decoder-capacity expansion.
  • Conclusion: The authors position the decoder, or decision part, as equally important as the encoder, or perception part, in end-to-end autonomous driving.They hope this exploration encourages further work on decoder capacity.

2. Related Work

Related work spans end-to-end driving, BEV representations, and coarse-to-fine prediction. ThinkTwice differs by jointly exploiting encoder capacity and using predicted behavior to retrieve critical features and anticipate the future for refinement.

  • End-to-end Autonomous Driving: End-to-end driving methods map sensor observations directly to actions or trajectories, with prior work including direct control, speed prediction, privileged experts, and joint control-planning models.The surveyed methods use learning-based, reinforcement-learning, and multi-task approaches across different system designs.
  • End-to-end Autonomous Driving: Most prior end-to-end methods focus on the encoder while using a simple MLP- or GRU-based decoder for final planning.ThinkTwice instead explores increasing decoder capacity while fully exploiting encoder capacity.
  • BEV Representation for Autonomous Driving: BEV representations preserve ground-plane spatial relationships and are used for sensor fusion, joint perception-planning, and geometric reasoning.Prior BEV work covers detection, segmentation, lane detection, planning, semantics, and way-point decoding.
  • BEV Representation for Autonomous Driving: NEAT iteratively aggregates features and decodes semantic categories and way-points in BEV, but does not explicitly project perspective-view image features into BEV and uses a simple MLP decoder.ThinkTwice explicitly aligns image and LiDAR features in BEV before decoding.
  • Coarse-to-fine Prediction: Coarse-to-fine strategies first generate coarse proposals and then extract proposal-based features for refined predictions across detection, optical flow, salient-object detection, and trajectory prediction.In autonomous-driving planning, LAV also iteratively refines way-points using the original feature with a simple RNN.
  • Coarse-to-fine Prediction: ThinkTwice uses coarse predictions to retrieve features in critical regions and anticipate the future, rather than refining solely from the original feature.This distinguishes its planning refinement mechanism from the described LAV approach.

3. Approach

ThinkTwice combines a BEV encoder with a cascaded decoder that predicts coarse actions and trajectories, imagines action-conditioned futures, retrieves goal-related features, and iteratively refines predictions. The encoder also receives feature-level supervision to provide denser decision-related learning signals.

  • 3.1. BEV Encoder: ThinkTwice transforms camera and LiDAR inputs into aligned BEV features, concatenates them, and processes the fused representation with 2D convolutions.Camera features use image backbones and LSS-based projection; LiDAR features use sparse 3D convolution and temporal point-cloud aggregation.
  • 3.1. BEV Encoder: The encoder receives dense feature-level supervision by matching middle BEV features to those of the privileged-input Roach teacher network.This gives each BEV grid decision-related supervision beyond sparse action labels.
  • 3.2.1 Coarse Prediction Module: A coarse prediction module uses environment and routing features to predict the ego vehicle’s initial action Ctrl0 and trajectory Traj0.The baseline flatten-plus-MLP design ignores spatial-temporal associations, motivating the subsequent modules.
  • 3.2.2 Look Module: The Look Module retrieves multi-scale camera and surrounding LiDAR features around predicted trajectory coordinates, producing a goal-related look feature.Camera coordinates are projected into the image plane for deformable attention, while LiDAR retrieves surrounding voxels; the look feature updates the environment representation.
  • 3.2.3 Prediction Module: The Prediction Module uses a spatial-GRU conditioned on coarse actions to generate future scene features, trained with teacher forcing and Roach BEV features.During training, ground-truth actions and trajectories provide inputs whose outputs are supervised using collected future-scene representations.
  • 3.2.4 Refinement Module: The refinement module predicts action and trajectory offsets from look, prediction, current-scene, and prior-output features, then updates the estimates additively.The cascaded module can be stacked, and multi-layer cascades yield notable performance gains.

4. Experiments

Experiments evaluate ThinkTwice in CARLA closed-loop benchmarks and analyze encoder, decoder, and capacity-design choices. The results support supervised geometric projection, stacked decoder modules, and the proposed encoder-decoder allocation.

  • Experimental Setup: CARLA evaluation uses the Town05 Long and Longest6 closed-loop benchmarks, with route completion, infraction score, and driving score as evaluation measures.Driving Score is the product of Route Completion and Infraction Score.
  • Comparison with SOTA: ThinkTwice achieves the best Driving Score on both benchmarks, while also achieving the best Infraction Score on Longest6.On Town05 Long, Roach and MILE have higher route completion but more collisions or traffic-rule violations, whereas Transfuser has the highest infraction score but lower route completion.
  • Experimental Setup: 189K frames from four towns are used for training, while Town05-Long is evaluated with three repeats and remains unseen during training.The data were collected in town01, town03, town04, and town06 at 2 Hz.
  • Encoder Design: Explicit depth and semantic segmentation during image-to-BEV projection substantially improves performance, whereas adding a second input frame yields only marginal improvement.The authors retain two frames because the decoder’s Prediction Module requires surrounding-agent motion clues.
  • Decoder Design: Five stacked decoder layers significantly improve results, while removing the Look Module or Prediction Module causes performance degradation.One additional decoder layer raises route completion but slightly lowers infraction score; the five-layer model demonstrates decoder scalability.
  • Model Capacity: Simply enlarging the encoder or classical MLP/GRU decoder does not bring performance gain, motivating capacity increases through the proposed supervised decoder paradigm.The encoder-size comparison reports a significant performance drop when TCP changes from ResNet-34 to ResNet-101.

5. Conclusion

ThinkTwice presents a scalable decoder paradigm with dense supervision and spatial-temporal priors. Stacking its decoder layers achieves state-of-the-art performance on two closed-loop autonomous-driving benchmarks.

  • Conclusion: ThinkTwice enlarges decoder capacity through scalable decoder layers with dense supervision and spatial-temporal priors.The proposed layers can be stacked to form the decoder.
  • Conclusion: Stacking the proposed decoder layers achieves state-of-the-art performance on two competitive closed-loop autonomous-driving benchmarks.The paper presents this result as the outcome of its scalable decoder design.

A. Implementation Details

The implementation section provides reproducibility details and states that the code and model will be made publicly available.

  • Implementation Details: The authors provide implementation details to facilitate reproduction and plan to make the code and model publicly available.The stated motivation is that end-to-end autonomous-driving models are large systems.

A.1. Data Collection

ThinkTwice uses multimodal expert driving logs collected from CARLA, including cameras, LiDAR, vehicle-state sensors, navigation commands, and auxiliary supervision signals.

  • Sensors and Expert: The data-collection setup uses four cameras, one LiDAR, IMU, GPS, and a speedometer, with Roach serving as the expert.The cameras cover front, left, right, and back views.
  • Collected Signals: Additional collected signals include target points, high-level commands, Roach’s value function, BEV feature maps, 1D features, and control actions.All raw data are converted into the ego coordinate system.

A.2. Models

The implementation uses established perception backbones and BEV processing, with specified training, augmentation, temporal-input, and prediction-horizon settings.

  • Model configuration: ResNet50 is used as the image backbone, while extra-data settings use ConvNext-base and PAFPN for multi-scale image features.LSS and depth code are adopted from prior work, and the semantic segmentation module uses a U-Net-like structure.
  • Model configuration: The BEV grid is 21x21, with two input frames normally and three frames for extra-data settings.The BEV scale is Front=30.4m, Back=-8.0m, Left=-19.2m, Right=19.2m.
  • Training: Training uses AdamW with learning rate 1e-4, cosine decay, effective batch size 128, weight decay 1e-7, and 60 epochs.Hidden dimensions are at most 256, and loss weights are tuned so each loss is approximately 1 at training onset.
  • Training: Image augmentation applies random color transformation and random cropping before BEV projection, stopping during the final ten epochs.The prediction time-horizon is 4, with gradient clipping at an L2-norm threshold of 35.

B. Discussion about Inference Computation

The discussion emphasizes that ThinkTwice has substantial inference computation and memory demands, primarily because it uses geometric fusion in BEV through LSS.

  • Inference computation: ThinkTwice has large MACs and GPU memory usage because it is the only compared model adopting geometric fusion in BEV with LSS.The authors choose BEV because it preserves spatial relationships on the ground plane and suits joint perception-planning and sensor fusion.
  • Inference computation: 1170G MACs are reported for ThinkTwice’s encoder, including 1157G from the LSS module.Table 7 defines MACs and GPU memory under inference mode and reports computational-burden comparisons.
  • Deployment constraints: Inference efficiency matters because autonomous-driving models typically run on edge devices with limited computational power and memory.More efficient BEV implementations and purpose-built edge devices and chips are identified as active directions for reducing this burden.

C. Visualization

The visualization compares predictions across decoder layers, with deeper-layer predictions shown as larger and brighter dots.

  • Visualization: Larger and brighter dots indicate predictions from deeper decoder layers.The visualization is intended to show the future-conditioned coarse-to-fine refinement process, which the authors describe as safer and smoother.
Loading 2305.06242v1…