Source-linked AI summary
PlanT: Explainable Planning Transformers via Object-Level Representations
Katrin Renz, Kashyap Chitta, Otniel-Bogdan Mercea, A. Sophia Koepke, Zeynep Akata, Andreas Geiger
TL;DR
Learning-based planners often rely on dense pixel-level representations, motivating the question of whether compact object-level inputs can support effective autonomous-driving planning. PlanT addresses this with imitation learning and a transformer architecture that represents vehicles and routes as objects. It achieves expert-level or better planning performance, faster inference, stronger sensor-based driving scores, and improved identification of relevant objects, while remaining evaluated in simulation and dependent on an imperfect expert.
Problem
Learning-based planners typically use dense, high-dimensional pixel-level scene representations, while it remains unclear whether such detail is necessary for effective planning.
Method
PlanT uses imitation learning with object-level vehicle and route inputs, a transformer encoder for waypoint prediction, and an auxiliary task predicting other vehicles’ futures.
Results
PlanT achieves expert-level or better CARLA planning performance, improves sensor-based driving score over prior methods, and identifies relevant objects more reliably than PlanCNN.
Takeaways & Limitations
Compact object-level representations can support high-performance, faster, and more explainable autonomous-driving planning in the evaluated CARLA settings.
Takeaways & Limitations
The experiments are conducted in simulation, use an imperfect imitation-learning expert, and include a perception experiment based on a single off-the-shelf module.
Abstract
from arXiv · showhide
Planning an optimal route in a complex environment requires efficient reasoning about the surrounding scene. While human drivers prioritize important objects and ignore details not relevant to the decision, learning-based planners typically extract features from dense, high-dimensional grid representations containing all vehicle and road context information. In this paper, we propose PlanT, a novel approach for planning in the context of self-driving that uses a standard transformer architecture. PlanT is based on imitation learning with a compact object-level input representation. On the Longest6 benchmark for CARLA, PlanT outperforms all prior methods (matching the driving score of the expert) while being 5.3x faster than equivalent pixel-based planning baselines during inference. Combining PlanT with an off-the-shelf perception module provides a sensor-based driving system that is more than 10 points better in terms of driving score than the existing state of the art. Furthermore, we propose an evaluation protocol to quantify the ability of planners to identify relevant objects, providing insights regarding their decision-making. Our results indicate that PlanT can focus on the most relevant object in the scene, even when this object is geometrically distant.
1 Introduction
PlanT replaces dense pixel-level scene representations with compact object-level inputs for transformer-based planning. On CARLA, the paper combines strong planning results with faster inference and explainability analysis.
- Motivation and approach: Most learned planners use high-dimensional pixel-level bird’s-eye-view representations, whereas PlanT represents scenes using nearby vehicles and route features.The paper investigates whether detailed pixel-level representations are necessary for convincing planning performance.
- Motivation and approach: PlanT uses an object-level representation with a transformer encoder and achieves state-of-the-art planning results despite its low feature dimensionality.The representation includes vehicle and route information rather than rendered detailed HD-map images.
- Evaluation: The study evaluates learning-based planning on the Longest6 benchmark in the CARLA simulator.The analysis examines planner design choices, performance, inference speed, and explainability.
- Explainability: PlanT’s attention weights support explanations of which objects influence planning decisions, and the paper introduces a protocol and metric for evaluating object prioritization.The protocol is intended to assess whether planners identify relevant objects for safe driving.
2 Related Work
Related work spans compact intermediate representations, transformer-based forecasting, and neural-network explainability. PlanT combines object-level inputs and transformer processing for full urban planning while adding a metric for explanation quality.
- Intermediate Representations for Driving: Affordance-based methods predict compact scene attributes before rule-based planning, but perform poorly on later CARLA benchmarks involving more complex tasks.The passage contrasts compact affordances with the higher demands of subsequent benchmarks.
- Transformers for Forecasting: Most motion-forecasting methods use rasterized inputs with CNNs, while later work combines object-level representations with graph neural networks or transformers.PlanT differs through architectural simplicity, its route representation, and closed-loop planning evaluation.
- Explainability: Self-driving explainability research has used text and heatmaps, whereas PlanT obtains post hoc explanations from learned attention.The paper also introduces a metric for measuring explanation quality in planning.
- Explainability: Unlike concurrent CAPO, which studies pedestrian–ego interactions on an empty route, PlanT evaluates full urban planning in dense traffic.PlanT’s evaluation also includes a metric for the quality of planner explanations.
3 Planning Transformers
PlanT uses imitation learning with object tokens for vehicles and route segments, a transformer encoder for waypoint prediction, and an auxiliary task forecasting other vehicles. Its training combines waypoint and vehicle-future losses.
- Task: The task is point-to-point urban navigation under traffic rules, trained by imitation learning to map object-level inputs to an expert’s future trajectory.The policy also receives the state of the next relevant traffic light.
- Scene representation: Scenes are tokenized as oriented vehicle and route-segment bounding boxes, each described by six attributes in bird’s-eye-view space.Vehicle speed is included, while route segments encode ordering, length, and lane width.
- Scene representation: Route segments are created by sampling route points, simplifying them with Ramer-Douglas-Peucker, and supplying a fixed number of segments to the policy.The segment length is clipped by Lmax.
- Token embeddings: Each token is linearly projected and combined with a learned type embedding indicating whether it represents a vehicle or route segment.The projection maps six-dimensional inputs into the model’s hidden dimension.
- Waypoint prediction: A transformer encoder aggregates token features through a learnable [CLS] token, which initializes an autoregressive decoder that predicts future waypoints alongside the traffic-light flag.The waypoint output is generated from the transformer representation and traffic-light input.
- Auxiliary task: PlanT adds a self-supervised auxiliary task that predicts future speed, position, orientation, and extent attributes for other vehicles.Each attribute is discretized into bins to represent uncertainty in multimodal futures.
- Training: Training uses a weighted multi-task combination of L1 waypoint loss and cross-entropy loss for future vehicle attributes.The weighting factor λ balances the primary and auxiliary objectives.
4 Experiments
The experiments evaluate PlanT’s driving performance, efficiency, sensor-based extension, and explainability on Longest6. Results show that object-level transformer planning benefits from richer inputs and scaling while identifying relevant vehicles effectively.
- Driving performance: PlanT reaches expert-level performance on Longest6 while requiring substantially less inference time than the evaluated baselines.Table 1 reports mean±std over three evaluations.
- Input representation: A full 360° view, route representation, and vehicle-speed cues substantially improve learned-planner driving scores.Including rear vehicles raises PlanT’s DS from 72.86 to 81.36, while removing speed input significantly reduces DS.
- Scaling: PlanT is more data-efficient than PlanCNN, and scaling from 1× to 3× data enables expert-level performance.PlanT reaches DS 81.36 in some runs versus expert DS 76.91; PlanT is also faster than PlanCNN.
- Training: The auxiliary self-supervised loss is crucial to PlanT’s performance without requiring additional annotation costs.The loss supervises multiple output tokens rather than only a single [CLS] token.
- Sensor-based planning: With an off-the-shelf perception module, PlanT outperforms TransFuser by 10.36 driving-score points and runs 2.7× faster.Inference time is 37.61 ms for PlanT with perception versus 101.24 ms for TransFuser.
- Explainability: PlanT’s relevance scores outperform distance-based selection and PlanCNN, achieving mean RFDS 96.82 versus 82.83 for PlanCNN.PlanT can identify dynamically relevant distant vehicles, though it sometimes focuses on a very close vehicle behind the ego vehicle.
5 Conclusion
PlanT combines a compact object-level representation with a transformer architecture to support efficient, high-performance, explainable autonomous-driving planning. Its results emphasize the importance of route encoding, full 360° context, vehicle speeds, scalable training, and careful interpretation of simulation and perception-based evaluations.
- PlanT uses an object-level representation and transformer architecture for efficient, high-performance, explainable autonomous-driving planning.
- The planner requires correctly encoded ego routes, a full 360° field of view, vehicle speeds, and sufficiently large architectures and datasets for state-of-the-art results.
- PlanT identifies the scene’s most relevant object using a metric and evaluation protocol designed to measure explainability.
- The evaluation is limited by imperfect expert demonstrations, simulation-only experiments, and a single off-the-shelf perception module that was not optimized for PlanT.