Perceive, Attend, and Drive: Learning Spatial Attention for Safe Self-Driving

I. INTRODUCTION

• Perception objectives weight all detected objects alike, including actors that may never interact with the vehicle, misaligning detection with safe driving.Source: §I · p. 1
• The proposed attention mask is learned for motion planning and reweights detection and forecasting losses toward objects that matter most.The weighting is data-driven rather than manually assigned.Source: §I · p. 1
• The attention-based model reduced collision rates, improved planning performance, and used much lower computational cost.Source: §I · p. 1
• Visualizing the attention mask is presented as a way to improve interpretability of end-to-end self-driving models.Source: §I · p. 1

II. RELATED WORK

• Prior attention work spans object recognition,,, instance segmentation, image captioning, question answering,, and NLP,,.Source: §II · p. 1
• In multi-task self-driving, perception, prediction, and planning use separate output branches with shared features, while this work uses one attention mask to optimize driving.The mask also weights instance-detection losses.Source: §II · p. 2
• Safety approaches include formal model checking,,,,, imitation learning,,,,, and learning risk-sensitive measures,.The authors report that sparse attention models experimentally avoid collisions better, although safety is not explicitly encoded in their loss.Source: §II · p. 2

III. PERCEIVE, ATTEND AND DRIVE

• SA-NMP takes LiDAR and HDMaps data and produces perception, prediction, and planning outputs.Source: Figure 3 · p. 2
• A U-Net generates attention that is applied to the input branches of residual blocks.Source: Figure 3 · p. 2
• The attention is learned toward planning, which is a direct model output.Source: Figure 3 · p. 2

A. A review on Neural Motion Planner (NMP)

• NMP jointly solves perception, prediction, and planning, using a shared backbone and separate task headers.Source: A. A review on Neural Motion Planner… · p. 2
• The backbone fuses voxelized LiDAR birds-eye-view features with rasterized HDMap features and downsamples spatial dimensions by four.It uses five blocks, with multi-scale features from the first four concatenated into the final block.Source: A. A review on Neural Motion Planner… · p. 2
• The planning header produces a cost volume over locations and a fixed future horizon, and inference chooses the lowest-cost sampled trajectory.Samples use mixtures of Clothoid, circle, and straight curves.Source: A. A review on Neural Motion Planner… · p. 2

B. Sparse Attention Neural Motion Planner (SA-NMP)

• The planner uses the same input representation and task headers as NMP, with a PnPNet backbone that fuses features across scales.Its cross-scale blocks use three parallel resolutions and residual connections; the final feature has 128 channels at 4× downsampled resolution.Source: B. Sparse Attention Neural Motion… · p. 3
• Binary attention gates the backbone to focus computation on regions relevant to planning a safe maneuver, enabling sparse convolution.The paper contrasts this with prior attention approaches that use road or vehicle masks and may spend computation on areas that do not affect the car.Source: B. Sparse Attention Neural Motion… · p. 3
• A small U-Net predicts a scalar score at each spatial location, binarized into an attention map applied residually to backbone features.The U-Net has skip connections and two downsample/upsample stages; the residual application is intended to avoid deteriorating features throughout the backbone.Source: B. Sparse Attention Neural Motion… · p. 3
• Gumbel softmax enables learning through the non-differentiable binary attention map, with a straight-through estimator using softmax in the backward pass.The softmax uses temperature constant K and represents the underlying soft attention.Source: B. Sparse Attention Neural Motion… · p. 3

C. Multi-Task Learning

• The end-to-end objective combines motion planning with perception and motion-forecasting losses in multi-task training.The stated objective also includes attention sparsity and standard weight decay terms.Source: C. Multi-Task Learning · p. 3
• The max-margin planning objective makes the human driving trajectory lower-cost than trajectories sampled by the model.The objective penalizes the maximum margin violation between the ground-truth trajectory and negative samples; traffic-rule violations are also represented in the penalty.Source: C. Multi-Task Learning · pp. 3–4
• Perception and forecasting use classification and regression losses, including binary cross-entropy and trajectory regression over future steps for boxes assigned to each spatial location.The regression targets predicted bounding-box shifts and instance trajectories relative to ground truth.Source: C. Multi-Task Learning · p. 4
• The spatial attention mask reweights auxiliary perception and forecasting losses so attended instances receive greater weight than all instances generally.The paper sets γ1 = 0.9 for attended instances and γ0 = 0.1 for all instances.Source: C. Multi-Task Learning · p. 4
• An ℓ1 attention loss encourages a focused, sparse mask, with λA controlling the sparsity strength.The total objective also includes standard weight decay.Source: C. Multi-Task Learning · pp. 3–4

IV. EXPERIMENTAL EVALUATION

• Drive4D training uses over 1 million frames from 5,000 scenarios, with validation on 5,000 frames from 500 scenarios.The evaluation uses both LiDAR and HD-maps.Source: §IV · p. 4
• The nuScenes v1.0 evaluation uses over 200,000 training frames and a 5,000-frame test set.The paper uses LiDAR only and omits HD-maps because of inaccurate localization.Source: §IV · p. 4
• The datasets differ in input scope: Drive4D uses LiDAR and HD-maps, while nuScenes uses LiDAR without HD-maps.The stated reason for omitting nuScenes HD-maps is their inaccurate localization.Source: §IV · p. 4

A. Implementation Details and Metrics

• Attention models start from pretrained dense SA-NMP backbone and header weights, then train for two epochs with Adam across 16 GPUs.Training uses batch size 5, an initial learning rate of 1 × 10−4, and decay by 0.1 at 1.0 and 1.6 epochs.Source: A. Implementation Details and Metrics · p. 4
• Planning L2, collision rate, and lane violation rate measure trajectory distance, collisions with other actors, and lane-boundary violations over six future timesteps.Lane violations are not evaluated on nuScenes because of inaccurate localization.Source: A. Implementation Details and Metrics · p. 4
• Baselines are Road Mask, Vehicle Mask, Proximity Mask, and Dense, covering map-defined roads, detections, a radius around the ego vehicle, and no sparse attention.The masks are end-to-end trained using static attention priors.Source: A. Implementation Details and Metrics · p. 4

Quantitative results

• Across computational budgets, learned attention outperforms baselines in collision rate and planning L2, using ∼80% fewer FLOPs than Dense SA-NMP with a 95% sparse mask.The comparison varies backbone depth and width; the reported FLOP reduction is relative to Dense SA-NMP.Source: B. Results · p. 4, Figure 4 · p. 5, Table I · p. 5

Qualitative results

• Attention shifts among nearby and distant vehicles and road markings; in Fig. 5 columns A/B, learned trajectories avoid future conflicts that baselines hit.Open-loop rear-end collisions remain, and the model under-attends the open road to its right, which could allow a right turn.Source: B. Results · pp. 4–5, Figure 5 · p. 6

Sparsity of learned attention

• Planning performance improves as attention sparsity increases, peaking at 95% sparsity with other settings held constant.Table II varies λA, the weight on the attention-mask ℓ1 regularization term.Source: B. Results · p. 5, Table II · p. 5

Perception and prediction loss reweighting

• Increasing γ1 relaxes detection-performance pressure on all actors, but at γ1 = 1.0 the model can avoid all actors and lose its PnP learning signal.Other settings, including sparsity at 95%, are held constant.Source: B. Results · p. 5, Table III · p. 5

Detection performance

• The 95% sparse learned-attention model detects actors within its attended region better than the dense model on the same actor subsets.Table IV compares detection on all actors and on actors inside the learned model's attention mask.Source: B. Results · p. 6, Table IV · p. 6

• Sparse attention masks gate computation in the feature backbone of an end-to-end self-driving network.Source: §V · p. 6
• Unlike attention focused on perception, the masks are optimized for motion planning, enabling better planned trajectories with greater efficiency and sparsity.Source: §V · p. 6
• Future work could extend the attention module with recurrent feedback from output layers to better leverage temporal information.Source: §V · p. 6