Source-linked AI summary
NEAT: Neural Attention Fields for End-to-End Autonomous Driving
Kashyap Chitta, Aditya Prakash, Andreas Geiger
TL;DR
Autonomous driving models need efficient representations of scenes’ semantic, spatial, and temporal structure, but camera-to-BEV feature association is difficult. NEAT learns query-dependent attention fields that compress image features for joint waypoint and semantic prediction. In challenging CARLA conditions, it outperforms strong baselines, matches the privileged expert on internal routes, and provides interpretable attention maps.
Problem
Camera-based BEV semantic prediction requires associating each spatiotemporal BEV query with relevant image pixels while reasoning about geometry, motion, intention, and scene interactions.
Method
NEAT uses an MLP query function and iterative attention maps to compress high-dimensional image features into compact representations for joint waypoint-offset and BEV-semantic prediction.
Results
NEAT outperforms several strong baselines and matches the privileged expert’s performance on internal CARLA routes in an evaluation with multiple towns, environmental conditions, and pre-crash scenarios.
Takeaways & Limitations
NEAT provides a flexible, compact intermediate representation with interpretable attention maps for end-to-end imitation-learning driving.
Takeaways & Limitations
Experiments focus on a single input frame (T = 1), and required road labels can come from LiDAR-derived ground planes or localized HD maps.
Abstract
from arXiv · showhide
Efficient reasoning about the semantic, spatial, and temporal structure of a scene is a crucial prerequisite for autonomous driving. We present NEural ATtention fields (NEAT), a novel representation that enables such reasoning for end-to-end imitation learning models. NEAT is a continuous function which maps locations in Bird's Eye View (BEV) scene coordinates to waypoints and semantics, using intermediate attention maps to iteratively compress high-dimensional 2D image features into a compact representation. This allows our model to selectively attend to relevant regions in the input while ignoring information irrelevant to the driving task, effectively associating the images with the BEV representation. In a new evaluation setting involving adverse environmental conditions and challenging scenarios, NEAT outperforms several strong baselines and achieves driving scores on par with the privileged CARLA expert used to generate its training data. Furthermore, visualizing the attention maps for models with NEAT intermediate representations provides improved interpretability.
1. Introduction
NEAT addresses the difficulty of associating camera features with spatiotemporal BEV locations by learning query-dependent attention fields. It jointly supports waypoint planning and BEV semantic prediction, achieving strong performance and interpretability in a challenging CARLA evaluation.
- Motivation: Autonomous driving requires representations that capture both spatial and temporal scene structure.Behavior cloning agents increasingly use auxiliary objectives to model this structure.
- Challenge: BEV semantic prediction improves intermediate representations but requires associating each spatiotemporal query with relevant image pixels.This association involves 3D geometry, scene motion, ego-motion, intention, and interactions between scene elements.
- Approach: NEAT uses an MLP query function to map locations (x, y, t) to attention maps and compact features without attention supervision.The compact representation supports dense prediction in space and time while retaining interpretable attention maps.
- Approach: NEAT predicts waypoint offsets for trajectory planning and BEV semantics as an auxiliary task from a shared representation.The architecture combines the NEAT feature representation with an implicit decoder for joint prediction.
- Evaluation: NEAT outperforms several strong baselines and matches the privileged expert’s performance on internal CARLA routes.The evaluation includes multiple towns, new environmental conditions, and challenging pre-crash traffic scenarios.
- Evaluation: The challenging CARLA evaluation setting contains 6 towns and 42 environmental conditions, with attention visualizations providing insights into learned driving behavior.The paper reports competitive secret-route scores with significantly fewer infractions than existing methods.
2. Related Work
Related work establishes compact neural implicit representations and BEV-based driving methods as relevant foundations. NEAT differs by learning query-dependent attention-based feature association for dynamic, camera-based driving.
- Implicit Scene Representations: Neural implicit representations provide compact, high-resolution scene representations through neural function approximators with constant memory footprints.Prior work applies them to geometry, texture, dynamics, lighting, and larger dynamic scenes.
- Implicit Scene Representations: NEAT uses neural approximator compactness to learn intermediate features for downstream driving rather than directly representing scene geometry.
- End-to-End Autonomous Driving: NEAT is applied to imitation-learning-based autonomous driving, while its encoder could also support affordance and reinforcement-learning methods.
- BEV Semantics for Driving: BEV representations expose 3D layout without object occlusion and correlate more directly with vehicle kinematics than projective image coordinates.
- BEV Semantics for Driving: Prior camera-based BEV driving methods use explicit camera-intrinsics-based projection or predict semantics only for static scenes, whereas NEAT learns attention-based association and includes time.
3. Method
NEAT jointly learns dense waypoint offsets and BEV semantics from image-derived features by querying spatiotemporal scene locations and iteratively refining attention. Its compact query-specific representation supports waypoint generation and controller-based driving.
- Waypoint and semantic targets: Waypoints are defined as future vehicle positions in the ego vehicle’s BEV coordinate system and form a trajectory over a fixed prediction horizon.The current vehicle is at (0, 0), with forward aligned to the positive y-axis.
- Waypoint and semantic targets: BEV semantic prediction densely labels arbitrary spatiotemporal query locations for both observed and future time-steps.The task uses a bounded spatial range and predicts scene dynamics alongside waypoints.
- Waypoint and semantic targets: Dense offset prediction maps each query point p = (x, y, t, x′, y′) to a 2D vector from the query location to the waypoint, conditioned on the target route location.Target locations provide driver intention when multiple future trajectories are plausible.
- Architecture: NEAT iteratively converts query points and encoded features into attention maps, whose softmax-weighted aggregation produces compact query-specific features for decoding.The decoder predicts semantic classes and waypoint offsets at each attention iteration, with intermediate predictions supervised during training.
- Training and control: Training samples sparse spatiotemporal query points and jointly optimizes semantic classification and waypoint-offset predictions across attention iterations.Earlier-iteration predictions are down-weighted by γ_i, while intermediate losses are reported to improve performance.
- Training and control: At test time, NEAT predicts offsets on a G × G grid, averages shifted query locations into future waypoints, and feeds waypoint geometry and red-light predictions to PID controllers.Longitudinal control uses waypoint-step magnitudes, while lateral control uses waypoint orientation.
4. Experiments
NEAT is evaluated in CARLA across internal and secret routes, challenging environments, and ablations against multiple baselines. It achieves expert-level internal driving performance, competitive leaderboard results, and interpretable, location-specific attention.
- Evaluation Setting: The evaluation covers navigation in CARLA 0.9.10 with dynamic agents, traffic rules, dangerous scenarios, multiple towns, and varied weather conditions.Internal evaluation uses 42 routes from 6 towns; the official Leaderboard contains 100 secret routes with unknown environmental conditions.
- Metrics: RC measures route completion, IS applies multiplicative penalties for infractions, and DS weights RC by IS.Reported metrics include Route Completion, Infraction Score, and Driving Score.
- Conditioning: BEV target conditioning and waypoint prediction improve adaptation to driver intention, producing smooth trajectories toward the supplied target location.NEAT varies predicted semantics and waypoint offsets across target locations and time steps.
- Driving Performance: NEAT obtains the best driving score on the 42 internal routes, matching or sometimes outperforming the expert used for data collection.Results report the mean and standard deviation over three evaluations for each model.
- Driving Performance: On the Leaderboard, NEAT is not best in DS but has the highest IS among the top three methods, indicating the safest driving behavior.The comparison uses 100 secret routes, while evaluation statistics were limited by the monthly evaluation budget.
- Runtime and Ablation: NEAT’s default model runs in real time at 30.37 ms per frame and reaches DS = 65.10, versus DS = 33.63 for the AIM-NEAT hybrid.Without the compute-intensive red-light indicator, NEAT runs at 18.60 ms; the comparison attributes the performance difference to attention maps and location-specific features.
5. Conclusion
NEAT advances interpretable, high-performance end-to-end autonomous driving by jointly addressing BEV semantic prediction and vehicle trajectory planning from camera inputs. In CARLA, it achieves the highest safety among state-of-the-art methods.
- NEAT jointly performs BEV semantic prediction and vehicle trajectory planning from camera inputs.
- The approach advances interpretable, high-performance end-to-end autonomous driving.
- NEAT achieves the highest safety among state-of-the-art methods on the CARLA simulator.