Source-linked AI summary
ResWorld: Temporal Residual World Model for End-to-End Autonomous Driving
Jinqing Zhang, Zehua Fu, Zelin Xu, Wenying Dai, Qingjie Liu, Yunhong Wang
TL;DR
Existing driving world models improve planning but redundantly model static regions and do not deeply use future scene representations to refine trajectories. ResWorld models dynamic objects with temporal residuals and uses FGTR to interact prior trajectories with future BEV features. It achieves state-of-the-art planning performance on nuScenes and NAVSIM, while potential dynamic objects such as pedestrians and parked cars remain a limitation.
Problem
World models redundantly model static regions, weakly interact with trajectories, and can collapse when future scene representations lack auxiliary supervision.
Method
ResWorld uses temporal residuals to model dynamic objects and FGTR to refine prior trajectories through interaction with future BEV features.
Results
ResWorld achieves state-of-the-art planning performance on nuScenes and NAVSIM, including 88.3% PDMS on NAVSIM.
Takeaways & Limitations
Temporal-residual modeling focuses future prediction on dynamic-object spatial distributions, while trajectory interaction also provides sparse supervision for future BEV features.
Takeaways & Limitations
TR-World cannot adequately capture potential dynamic objects such as pedestrians and parked cars through temporal residuals.
Abstract
from arXiv · showhide
The comprehensive understanding capabilities of world models for driving scenarios have significantly improved the planning accuracy of end-to-end autonomous driving frameworks. However, the redundant modeling of static regions and the lack of deep interaction with trajectories hinder world models from exerting their full effectiveness. In this paper, we propose Temporal Residual World Model (TR-World), which focuses on dynamic object modeling. By calculating the temporal residuals of scene representations, the information of dynamic objects can be extracted without relying on detection and tracking. TR-World takes only temporal residuals as input, thus predicting the future spatial distribution of dynamic objects more precisely. By combining the prediction with the static object information contained in the current BEV features, accurate future BEV features can be obtained. Furthermore, we propose Future-Guided Trajectory Refinement (FGTR) module, which conducts interaction between prior trajectories (predicted from the current scene representation) and the future BEV features. This module can not only utilize future road conditions to refine trajectories, but also provides sparse spatial-temporal supervision on future BEV features to prevent world model collapse. Comprehensive experiments conducted on the nuScenes and NAVSIM datasets demonstrate that our method, namely ResWorld, achieves state-of-the-art planning performance. The code is available at https://github.com/mengtan00/ResWorld.git.
1 INTRODUCTION
World models improve end-to-end driving by predicting future scenes, but redundant static-scene modeling and weak trajectory interaction limit their effectiveness. ResWorld addresses these issues with temporal-residual dynamic-object modeling and Future-Guided Trajectory Refinement.
- World models use future scene prediction as a proxy task to improve driving-scene understanding and planning accuracy.
- Static regions can be retained across future scenarios, whereas dynamic vehicles and pedestrians require more precise modeling.
- TR-World extracts dynamic-object information from temporal residuals of aligned scene queries without auxiliary perception tasks.
- FGTR interacts prior trajectories with predicted future BEV features to detect collisions or drivable-area deviations and refine planning.
- ResWorld achieves state-of-the-art planning results on nuScenes and NAVSIM benchmarks.
- ResWorld represents future BEV features in the current coordinate system, avoiding redundant modeling of static objects.
2 RELATED WORKS
Related work presents end-to-end driving systems that integrate perception and planning, increasingly using world models for spatial-temporal scene understanding. Existing world-model approaches include occupancy-centric modeling and driving-video generation.
- End-to-end systems predict trajectories directly from raw sensor data through integrated models for cost-effective autonomous driving.
- Earlier approaches combine modules such as map perception, BEV occupancy, detection, tracking, motion prediction, and planning.
- World-model-based systems replace auxiliary tasks while improving spatial-temporal scene understanding for planning.
- Occ-World uses unified occupancy-centric world modeling, whereas Drive-WM generates driving videos through joint spatial-temporal modeling.
3 METHOD
ResWorld builds prior trajectories from temporally aligned BEV features, then models dynamic-object changes through temporal residuals and refines planning using predicted future BEV features. Its framework avoids redundant static-object prediction while coupling future-scene modeling to trajectory refinement.
- Prior Trajectory Prediction: High-quality BEV features from multiple timestamps are aligned to the current coordinate system and fused for temporal scene representation.The current BEV feature and past features are spatially transformed before fusion, supporting temporal residual extraction.
- Prior Trajectory Prediction: Sparse scene queries are extracted from fused BEV features, and waypoint queries decode prior ego trajectories with one row per future timestamp.TokenLearner produces sparse scene queries, while an MLP decodes trajectory coordinates from waypoint queries.
- Temporal Residual World Model: Temporal residuals between aligned scene queries isolate changes across timestamps, providing information about dynamic objects.Residuals are calculated by subtracting scene queries from adjacent timestamps.
- Temporal Residual World Model: TR-World predicts future dynamic-object distributions from temporal residuals and combines them with fused static-scene information to form future BEV features.The model accumulates residual information across timestamps and expands it over Bfuse using TokenFuser.
- Future-Guided Trajectory Refinement: FGTR uses prior trajectories as reference points for deformable attention between waypoint queries and future BEV features, producing refined final trajectories.The interaction checks future environmental information around the ego vehicle for collisions and drivable-area violations.
- Future-Guided Trajectory Refinement: Spatial and temporal supervision from trajectory reference points and waypoint timestamps encourages future BEV features to represent cross-temporal dynamic-object information and prevents world-model collapse.Training uses L1 losses on prior and final trajectories, while future BEV features are not directly supervised with real future data.
4 EXPERIMENT RESULS
ResWorld achieves state-of-the-art planning performance on nuScenes and NAVSIM, with complementary evaluations showing benefits from TR-World, FGTR, and their interaction. Ablations indicate improved dynamic-object prediction, trajectory quality, and resistance to world-model collapse.
- ResWorld achieves state-of-the-art planning accuracy on nuScenes and NAVSIM benchmarks.
- NAVSIM: 89.0% PDMS is achieved on NAVSIM with historical frame data, while the no-history configuration reaches 88.3% PDMS.
- Component efficiency: The combined TR-World and FGTR modules reduce baseline average L2 error by 8.4% without ego status and 9.2% with ego status.
- Component efficiency: The combined modules reduce baseline average collision rate by 25.8% without ego status and 39.3% with ego status.
- Future-Guided Trajectory Refinement: FGTR interacts with prior trajectories and future BEV features to produce more spatially diverse predictions and prevent world-model collapse.
- Temporal Residual World Model: TR-World predicts more accurate future BEV features than a normal world model by focusing on dynamic objects rather than redundant static-object modeling.
- Prior trajectory: Prior trajectories improve despite using the baseline architecture, because TR-World and FGTR optimize the BEV features used for planning.
- Qualitative results: Qualitative planning results show ResWorld trajectories avoiding collisions with vehicles and curbs more effectively than SSR.
5 CONCLUSION
ResWorld combines temporal-residual dynamic-object modeling with future-guided trajectory refinement to improve planning while reducing redundant static-region modeling. However, TR-World cannot adequately capture some potential dynamic objects, which remain handled by the prior trajectory branch.
- Conclusion: ResWorld uses temporal residuals to focus world-model forecasting on dynamic objects and avoid redundant modeling of static regions.The framework extracts dynamic-object information from changes in scene representations.
- Conclusion: Future-guided trajectory refinement uses future BEV features to correct prior trajectories and supplies sparse spatial-temporal supervision that alleviates world-model collapse.The interaction also reduces the probability of driving accidents according to the paper’s conclusion.
- Conclusion: ResWorld achieves state-of-the-art planning performance on both nuScenes and NAVSIM benchmarks.
- Limitations and Future Work: TR-World cannot adequately capture potential dynamic objects such as pedestrians and parked cars through temporal residuals.These objects are processed alongside static objects by the prior trajectory prediction branch.