PaperlayerView exact revision on arXiv

Source-linked AI summary

ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning

Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, Dacheng Tao

arXiv:2207.07601v2cs.CV

TL;DR

Autonomous-driving systems often use discrete pipelines, while interpretable vision-based end-to-end systems remain comparatively underexplored. ST-P3 jointly learns spatial-temporal features for perception, prediction, and planning, and reports state-of-the-art results on nuScenes and CARLA.

  • Problem

    The paper asks how far perception, prediction, and planning performance can improve in an interpretable vision-based end-to-end framework.

  • Method

    ST-P3 uses egocentric aligned 3D accumulation, dual-pathway temporal modeling, and prior-knowledge trajectory refinement for vision-based driving.

  • Results

    ST-P3 achieves state-of-the-art performance on benchmarks from nuScenes and the CARLA simulator.

  • Takeaways & Limitations

    The study provides a systematic analysis of an interpretable end-to-end vision-based autonomous-driving pipeline across perception, prediction, and planning.

Abstract

from arXiv · show

Many existing autonomous driving paradigms involve a multi-stage discrete pipeline of tasks. To better predict the control signals and enhance user safety, an end-to-end approach that benefits from joint spatial-temporal feature learning is desirable. While there are some pioneering works on LiDAR-based input or implicit design, in this paper we formulate the problem in an interpretable vision-based setting. In particular, we propose a spatial-temporal feature learning scheme towards a set of more representative features for perception, prediction and planning tasks simultaneously, which is called ST-P3. Specifically, an egocentric-aligned accumulation technique is proposed to preserve geometry information in 3D space before the bird's eye view transformation for perception; a dual pathway modeling is devised to take past motion variations into account for future prediction; a temporal-based refinement unit is introduced to compensate for recognizing vision-based elements for planning. To the best of our knowledge, we are the first to systematically investigate each part of an interpretable end-to-end vision-based autonomous driving system. We benchmark our approach against previous state-of-the-arts on both open-loop nuScenes dataset as well as closed-loop CARLA simulation. The results show the effectiveness of our method. Source code, model and protocol details are made publicly available at https://github.com/OpenPerceptionX/ST-P3.

1 Introduction

ST-P3 addresses the limitations of modular and implicit autonomous-driving pipelines with an interpretable vision-based end-to-end framework that jointly improves spatial-temporal features for perception, prediction, and planning. It uses aligned 3D feature accumulation, dual-pathway temporal modeling, and prior-knowledge refinement, and is evaluated on nuScenes and CARLA.

  • End-to-end learning directly maps raw sensor data to planning routes or control signals, allowing feature representations to be optimized toward the system’s ultimate goals.
  • Vision-based interpretable pipelines remain less explored than LiDAR-based approaches, motivating systematic investigation of perception, prediction, and planning together.
  • ST-P3 transforms camera features into 3D and accumulates past features in the current egocentric coordinate system before BEV transformation to preserve geometry.
  • A dual pathway models historical motion variations alongside temporal prediction to strengthen future semantic representations, while refinement uses prior features, high-level commands, and no HD maps for planning.
  • ST-P3 reports state-of-the-art performance on nuScenes and CARLA and provides its codebase and protocol details publicly.

2 Related Work

Related work spans interpretable end-to-end systems, BEV representation, future motion prediction, and motion planning. ST-P3 builds on explicit representations while addressing vision-to-BEV transformation and integrated planning without HD maps.

  • Interpretable End-to-End Framework: Interpretable end-to-end systems primarily use LiDAR and HD maps to expose intermediate representations and support safety-oriented analysis.
  • Bird’s Eye View Representation: BEV representations preserve 3D scene layout and reduce occlusion and scale-distortion issues, but projecting camera features into BEV remains non-trivial.
  • Bird’s Eye View Representation: ST-P3 follows depth-based camera-to-BEV projection while accumulating aligned past 3D features rather than transforming past features frame by frame.
  • Motion Prediction: Existing motion-prediction methods often rely on ground-truth perception and HD maps, whereas end-to-end approaches increasingly use raw sensors for future prediction.
  • Motion Planning: Implicit planners directly generate trajectories or controls but can lack robustness and interpretability; explicit planners select trajectories using cost maps and samplers.

3 Methodology

ST-P3 builds interpretable spatiotemporal features for perception, prediction, and planning by combining 3D ego-centric alignment, dual-pathway prediction, and prior-knowledge trajectory refinement.

  • Perception: ST-P3 extracts sequential camera features, lifts them into 3D with depth prediction, and fuses them spatially and temporally in BEV space.The framework uses egocentric aligned accumulation to align past features with the current view before aggregation.
  • Perception: Egocentric alignment transforms past feature cubes into the current SDV-centered coordinate system before pooling them into BEV feature maps.This addresses alignment across timestamps for downstream prediction and planning.
  • Perception: Temporal fusion combines aligned BEV features with self-attention for static objects and 3D convolution with ego-motion matrices for dynamic objects.The method uses the discount α = 0.5 in the self-attention accumulation and concatenates motion information for temporal fusion.
  • Prediction: The prediction module models future uncertainty with diagonal Gaussians and combines uncertainty-driven and historical-feature pathways recursively across future horizons.One pathway uses samples from η_t, while the other uses historical features and GRU recurrence to account for past motion variations.
  • Planning: The planner samples trajectories under a high-level command, selects the minimum-cost candidate using occupancy and map information, and refines it with front-view features.Its cost combines occupancy-related, vehicle-dynamics, and rule-based subcosts, while a GRU incorporates target-point and traffic-light information.

4 Experiments

ST-P3 is evaluated on open-loop nuScenes perception, prediction, and planning, plus closed-loop CARLA driving. It achieves strong benchmark performance, while ablations attribute gains to its spatial-temporal modules.

  • Evaluation Setup: ST-P3 evaluates open-loop performance on nuScenes and closed-loop applicability on CARLA, using 1.0s of past context and predicting 2.0s ahead.The closed-loop setting measures Route Completion and Driving Score, including collision penalties.
  • Perception: 2.51% higher mean BEV segmentation IoU than the previous state of the art is reported on the nuScenes validation set.The gain is attributed to Egocentric Aligned Accumulation.
  • Prediction: ST-P3 achieves the state of the art on all reported temporal segmentation metrics for future prediction.The evaluation includes IoU, PQ, RQ, and SQ; only previously observed vehicles are predicted.
  • Planning: ST-P3 achieves the lowest collision rate across all open-loop planning time intervals.The open-loop planning table compares collision rates across time intervals.
  • Closed-loop Planning: ST-P3 outperforms camera-based methods on all closed-loop metrics and is comparable with the LiDAR-based method.It shows strong route completion, including in long-range tests, although another method has a higher driving score on long routes.
  • Ablation Study: 1.31% improvement follows explicit depth supervision after Egocentric Aligned Accumulation in the perception ablation.The ablation reports a 0.79% gain from EAA alone and a 1.31% gain with explicit depth supervision.
  • Ablation Study: Dual Modelling and loss for all states improve vehicle IoU by 1.54% and vehicle PQ by 3.09%, while sampler-refinement combinations improve planning safety and accuracy.Removing front-view refinement or prior sampling knowledge produces high L2 error and collision rate.

5 Conclusions

The paper concludes that ST-P3 improves spatial-temporal feature learning for interpretable end-to-end vision-based autonomous driving. Its three modules together produce strong performance relative to prior state-of-the-art methods.

  • Conclusions: ST-P3 combines 3D egocentric feature accumulation, dual-pathway temporal modelling, and prior-knowledge refinement across driving tasks.These components target geometry preservation, probabilistic semantic representations across frames, and road-element information, respectively.
  • Conclusions: The combined pipeline achieves impressive performance compared with previous state-of-the-art methods.

A.1 Architecture for Perception and Prediction

The architecture transforms multi-camera features into temporally integrated BEV representations, models future uncertainty, and scores then refines candidate trajectories.

  • Perception: Camera images are encoded into front-view features, lifted into 3D using predicted depth, aligned across time with ego-motion, and sum-pooled into BEV maps.Historical information is integrated during temporal fusion before 3D-convolutional temporal processing.
  • Perception: Temporal fusion uses multiple 3D kernel sizes and pyramid pooling to account for different temporal receptive fields.
  • Prediction: Future uncertainty is modeled with Gaussian or Bernoulli distributions before temporal prediction of future BEV segmentation.The Gaussian branch estimates mean and log variance from the present feature.
  • Planning: Planning penalizes trajectories intersecting occupied regions, high-velocity uncertain occupancy, excessive proximity to a leading vehicle, and lane-line distance.The leading-vehicle cost depends on SDV velocity and is ineffective during lane changes.
  • Planning: A GRU refines the selected trajectory using front-camera encoder features, the current and selected positions, and the target point.

B Depth Supervision

The depth-supervision pipeline generates explicit nuScenes depth maps using semantic segmentation and a self-supervised depth method, then evaluates them with standard depth metrics.

  • Depth Supervision: Explicit depth supervision is prepared by generating depth maps for the nuScenes dataset.
  • Depth Supervision: FSRE-Depth uses semantic segmentation results as input and is trained with nuScenes front-view images to generate depth supervision.The segmentation model is first trained on Mapillary Vistas, then applied to nuScenes before FSRE-Depth training.
  • Depth Supervision: Depth maps are evaluated with absolute relative error, square relative error, root mean square error, root mean square logarithmic error, and threshold-1.25 accuracy.

C.1 Protocols

ST-P3 is evaluated in both open-loop nuScenes and closed-loop CARLA settings, using temporal image context to produce driving trajectories. The protocol includes qualitative perception outputs and simulator execution under limited camera coverage.

  • Evaluation settings: ST-P3 is evaluated open-loop on nuScenes and closed-loop in the CARLA simulator.The nuScenes dataset supports open-loop evaluation, while CARLA provides closed-loop demonstration.
  • nuScenes protocol: The nuScenes protocol uses 1.0s of past context to predict 2.0s of future context from seven frames.Each batch contains three past frames and four future frames.
  • CARLA protocol: CARLA training uses the remaining scenarios after reserving Town05 for evaluation, with trajectories executed until destination arrival or a time limit.The simulator receives 1.0s of past context and images from four cameras.
  • Perception outputs: The perception outputs include an original image, a segmentation result, and a predicted depth map.These are presented as the three components of Fig. 7.

C.2 Open-loop Experimental Results on nuScenes

The nuScenes qualitative results visualize ST-P3’s learned cost volume and composite semantic graph. Cars receive higher costs, while open drivable areas receive lower costs.

  • Qualitative visualizations: ST-P3 visualizes its learned cost volume alongside a composite graph of multiple semantic elements on nuScenes.The visualization presents both intermediate planning-related representations and semantic structure.
  • Cost-map encoding: Darker cost-map regions indicate smaller cost values, whereas lighter regions indicate larger values.This color interpretation is used throughout the qualitative nuScenes figures.
  • Qualitative pattern: Cars generally occupy higher-cost regions, while open drivable areas generally have lower costs.The qualitative results show this cost structure across the presented scenes.

C.3 Closed-loop Planning Results on CARLA Simulator

Closed-loop CARLA results show that ST-P3 can continue toward the expected track when vision-based map accuracy deteriorates off-center. The figures illustrate straight-road, pedestrian, lane-center, and intersection scenarios.

  • Off-center driving: When the vehicle deviates from lane centers, map detection accuracy drops significantly in CARLA.The authors associate this with training data being concentrated in normal circumstances.
  • Refined planning: Despite low map accuracy, ST-P3’s refinement operation keeps the vehicle traveling toward the expected track.The traditional sampler and cost-map method instead generally deviates from the expected track behavior in this condition.
  • Pedestrian response: One scenario shows ST-P3 predicting a slowing-down trajectory after detecting a pedestrian.This figure provides a pedestrian-specific closed-loop planning example.
  • Centerline driving: The BEV representations become normal when driving on a predetermined centerline trajectory.This behavior is illustrated in the CARLA qualitative results.
Loading 2207.07601v2…