Source-linked AI summary
ResAD: Normalized Residual Trajectory Modeling for End-to-End Autonomous Driving
Zhiyu Zheng, Shaoyu Chen, Haoran Yin, Xinbang Zhang, Jialv Zou, Xinggang Wang, Qian Zhang, Lefei Zhang
TL;DR
Directly predicting raw trajectories in E2E autonomous driving is challenged by spatio-temporal imbalance and risks spurious correlations. ResAD predicts deviations from an inertial reference and applies point-wise residual normalization; it reports 88.8 PDMS on NAVSIM v1 and best-or-second-best performance across almost all NAVSIM v2 extended sub-metrics.
Problem
Method
ResAD predicts residual deviations from a deterministic inertial reference and applies Point-wise Residual Normalization to address scale variance across trajectory points.
Source: §1 · p. 2, §3.2 · p. 4Results
ResAD reaches 88.8 PDMS on NAVSIM v1 with ResNet-34, surpassing DiffusionDrive and WoTE, and is best or second-best on almost all NAVSIM v2 extended sub-metrics.
Source: §4.3 · p. 6Takeaways & Limitations
ResAD is presented as a more robust, stable, and scalable foundation for E2EAD systems.
Source: §5 · p. 9
Abstract
from arXiv · showhide
End-to-end autonomous driving (E2EAD) systems, which learn to predict future trajectories directly from sensor data, are fundamentally challenged by the inherent spatio-temporal imbalance of trajectory data. This imbalance creates a significant optimization burden, causing models to learn spurious correlations instead of robust driving logic, while also prioritizing uncertain, distant predictions, thereby compromising immediate safety. To address these issues, we propose ResAD, a novel Normalized Residual Trajectory Modeling framework. Instead of predicting the future trajectory directly, our approach reframes and simplifies the learning task by predicting the residual deviation from a deterministic inertial reference. This inertial reference serves as a strong physical prior, compelling the model to move beyond simple pattern-matching and instead focus its capacity on learning the necessary, context-driven deviations (e.g., traffic rules, obstacles) from this default, inertially-guided path. To mitigate the optimization imbalance caused by uncertain, long-term horizons, ResAD further incorporates Point-wise Normalization of the predicted residual. This technique re-weights the optimization objective, preventing large-magnitude errors associated with distant, uncertain waypoints from dominating the learning signal. On the NAVSIM v1 and v2 benchmarks, ResAD achieves state-of-the-art results of 88.8 PDMS and 85.5 EPDMS with only two denoising steps, demonstrating that ResAD significantly simplifies the learning task and improves planning performance. The code will be released to facilitate further research.
1. Introduction
Direct trajectory prediction in E2EAD can foster spurious correlations and let uncertain distant-waypoint errors outweigh near-term safety. ResAD instead learns residual deviations from an inertial reference and applies point-wise normalization to address these challenges.
1. Introduction
- Directly predicting raw trajectories may make models learn spurious correlations instead of robust safe-driving logic.The paper gives the example of a model associating braking with a lead vehicle’s brake lights but missing the red light that caused it to stop.Source: §1 · p. 2
- Because distant waypoints are more uncertain, their large training losses can outweigh precision on the near-term path needed for immediate collision avoidance.The paper calls this the planning horizon dilemma.Source: §1 · p. 2
- ResAD combines an inertial reference with learned residual deviations and Point-wise Residual Normalization to simplify trajectory learning and mitigate optimization imbalance.The inertial reference represents the vehicle’s default path without active control; residuals encode necessary deviations, while normalization prevents high-magnitude residuals from dominating the learning signal.Source: §1 · p. 2
- 88.8 PDMS and 85.5 EPDMS are the reported state-of-the-art scores for ResAD on the NAVSIM v1 and v2 benchmarks.The reported benchmark scores are presented as evidence for the framework’s effectiveness.Source: §1 · p. 2
2. Related Work
Prior E2EAD work developed representations, sensor fusion, architectures, and generative planning, while most methods still predict trajectories directly. ResAD represents trajectories through an inertial reference and learned residuals, and uses perturbed references for context-aware multimodal planning without a fixed vocabulary.
2.1. End-to-End Autonomous Driving
- ResAD departs from direct future-trajectory prediction by decomposing trajectories into a physics-based inertial reference and a learnable residual.The paper presents this representation as more structured and interpretable.Source: §2.1 · p. 2
- Prior E2EAD research explored improved representations, sensor fusion, and architectures, but predominantly retained direct trajectory prediction.The paper situates its approach against this shared prediction paradigm.Source: §2.1 · p. 2
2.2. Multimodal Planning
- Several multimodal planners use static trajectory vocabularies, which can require evaluating irrelevant options and restrict outputs to predefined candidates.The related-work discussion names Hydra-MDP, GTRS, and DiffusionDrive among vocabulary-based approaches.Source: §2.2 · p. 3
- ResAD generates context-aware multimodal trajectories by denoising from Gaussian noise, while its framework conditions DiffDecoder on perturbed inertial references.The figure identifies the first- and fifth-ranked output trajectories as Top-1 Traj and Top-5 Traj.Source: §2.2 · p. 3, Figure 2 · p. 3
3. Methodology
ResAD uses a diffusion framework to predict normalized trajectory residuals rather than future points directly. It combines an inertial reference, point-wise residual normalization, perturbed references for multimodal predictions, and a ranker that selects among candidate trajectories.
3.1. Preliminaries
- ResAD uses a vanilla diffusion framework that adds noise over multiple timesteps, trains a denoiser to predict clean samples from noisy ones, and iteratively denoises at inference.The model conditions denoising on information c; the section defines the data samples as normalized residuals.Source: §3.1 · p. 3
3.2. Normalized Residual Trajectory Modeling
- ResAD uses a constant-velocity inertial reference and predicts the residual needed to reach the ground-truth trajectory, framing prediction as learning control interventions.The residual is the point-wise difference between the ground-truth trajectory and the no-control reference.Source: §3.2 · p. 4
- Point-wise Residual Normalization addresses residual scale variation across the horizon, which can let far-field errors dominate optimization over near-field adjustments.It applies component-wise min-max scaling using extrema precomputed across training trajectories and timesteps.Source: §3.2 · p. 4
- Perturbing the inertial reference generates diverse intent hypotheses and requires the model to handle ego-sensor noise.The method perturbs initial velocity to create distinct references and corresponding residuals.Source: §3.2 · p. 4
- The diffusion decoder denoises normalized residuals conditioned on encoded features and positional encodings from perturbed references, with inference using two DDIM timesteps.The reference encodings help distinguish intent hypotheses; predicted residuals are added to their corresponding references to form multimodal trajectories.Source: §3.2 · pp. 4–5
3.3. Multimodal Trajectory Ranker
- The trajectory ranker combines candidate trajectories with environment representations and ego status, predicts scores for PDMS or EPDMS metrics, and selects the highest weighted score.Its training distills scores from the rule-based planner and ground-truth waypoints.Source: §3.3 · p. 5
4. Experiments
Experiments evaluate ResAD on NAVSIM benchmarks and compare its planning performance, component contributions, and integration across planners. The results include strong benchmark scores and improvements associated with residual modeling, normalization, and inertial-reference perturbation.
4.1. Benchmark
- NAVSIM uses NuPlan sensor data sampled at 2 Hz, with 1,192 NAVTRAIN and 136 NAVTEST scenarios; trajectories are simulator-validated.Source: §4.1 · pp. 5–6
4.2. Implementation Details
- ResAD uses two cascaded diffusion decoder layers and DDIM inference with two denoising steps, then a trained ranker selects the output trajectory.It trains for 100 epochs on NAVTRAIN and predicts eight timesteps at 0.5-second intervals.Source: §4.2 · p. 6
4.3. Main Results
- 88.8 PDMS with ResNet-34 exceeds DiffusionDrive (88.1) and WoTE (88.3); ResAD is best or second-best across almost all NAVSIM v2 extended metrics.With V2-99, ResAD reaches 90.6 PDMS; versus DiffusionDrive, EP is 88.2 vs. 87.5 and DAC is 97.2 vs. 95.9.Source: §4.3 · p. 6, Table 2 · p. 6
- ResAD directly explores plausible, context-consistent behaviors, while DiffusionDrive’s static vocabulary can produce irrelevant or infeasible options that require filtering.The paper illustrates a straight-ahead proposal in a sharp turn as an example of an invalid option.Source: §4.3 · p. 6, Figure 3 · p. 7
4.4. Ablation Studies and Analysis
- TRM adds the key component gain: DAC +2.3, EP +2.5, and PDMS +1.2; PRNorm and IRP further raise performance, with IRP reaching 88.8 PDMS.The ranker alone adds PDMS +0.2; IRP adds +1.6 over M3, including EP +2.0 and DAC +0.8.Source: §4.4 · pp. 6–7
- PRNorm accelerates loss decline and yields higher per-step PDMS than vanilla min-max normalization.The authors attribute its optimization effect to preventing far-field errors from dominating.Source: §4.4 · p. 7
- Residuals are small when cruising and become directional in dynamic scenes, while IRP perturbs inertial baselines to expand the plausible behavior neighborhood.The described residuals point away from obstacles or follow road curvature; the combined IR and IRP focus capacity on necessary changes.Source: §4.4 · p. 8
- On Transfuser, TRM raises PDMS from 84.0 to 85.2 and TRM+PRNorm to 85.6; ResAD shows consistent performance and continuous growth as inference-time Kinfer increases.Comparable gains are observed on TransfuserDP. The approach requires limited changes to architecture or loss design.Source: §4.4 · p. 8, Figure 6 · p. 8
5. Conclusion
ResAD reframes E2EAD trajectory prediction through Trajectory Residual Modeling (TRM) and Point-wise Residual Normalization (PRNorm). The authors report state-of-the-art NAVSIM results and present the framework as a more robust, stable, and scalable foundation for E2EAD.
5. Conclusion
- ResAD combines TRM and PRNorm to address optimization imbalance and the risk of learning spurious correlations in raw-trajectory prediction.The authors report state-of-the-art results on NAVSIM and describe the framework as a more robust, stable, and scalable foundation for E2EAD systems.Source: §5 · pp. 8–9
- TRM uses an inertial reference as a physical prior so the model learns necessary, context-driven residuals rather than spurious patterns.The authors say this simplifies the task and focuses model capacity on robust driving logic.Source: §5 · p. 9
- PRNorm prioritizes safety-critical near-term adjustments over uncertain long-term predictions to mitigate optimization imbalance.Source: §5 · p. 9
- The authors report state-of-the-art results on NAVSIM and characterize ResAD as a robust, stable, and scalable foundation for E2EAD.Source: §5 · p. 9