Source-linked AI summary
RiskWorld: Object-Centric Latent World Modeling for Autonomous Driving Risk Identification
Jingzheng Li, Yufei Ge, Qianren Mao, Zhijun Chen, Bing Li, Xingyu Peng, Baochang Zhang, Xianglong Liu
TL;DR
Autonomous-driving risk identification must localize which observed object will become safety-critical, but existing methods often rely on scene-level prediction, indirect ego behavior, or post-hoc geometry. RiskWorld rolls relation-aware object states into the future with an object-centric latent world model and decodes object-level risk. On RiskBench, it reports 63.0% overall F1 and a 2.1% false-alarm rate, while analyses support early risk discrimination and planning relevance.
Problem
Driving risk identification must determine which observed participant or obstacle will become safety-critical to the ego vehicle as its relation evolves over time.
Method
RiskWorld combines predictive video representations and ego–object histories into relation-aware object states, rolls them forward with RSSM-style latent dynamics, and decodes object-level risk with auxiliary relation and temporal-risk predictions.
Results
63.0% overall F1 and a 2.1% false-alarm rate are reported on RiskBench, with analyses supporting early risk-source discrimination and planning relevance.
Takeaways & Limitations
Future ego–object relation rollout provides selective risk evidence beyond proximity, while selected objects retain planning-relevant information under filtered observation.
Takeaways & Limitations
Evaluation is limited to simulation, while complex multi-object interactions, long-tail behaviors, and severe occlusions remain open challenges.
Abstract
from arXiv · showhide
Autonomous driving risk identification aims to determine which observed object is likely to become safety-critical to the ego vehicle. Existing approaches typically predict scene-level accidents, infer risk objects indirectly from ego behavior, or apply geometric checks after trajectory forecasting, without directly using predicted ego--object relations for risk-source localization. We propose RiskWorld, an object-centric latent world model that identifies risk from the imagined evolution of each candidate relative to the ego vehicle. RiskWorld combines pretrained predictive video representations with structured ego--object histories, contextualizes observed interactions, and rolls relation-aware object states into the future using RSSM-style latent dynamics. It decodes the rollout into object-level risk scores, supported by auxiliary future-relation and temporal-risk predictions. Inference uses only observations up to the current time, while logged futures provide training supervision. On RiskBench, RiskWorld achieves the best overall F1 of 63.0\% and the lowest false-alarm rate of 2.1\%. Further analyses show that the learned rollout captures the evolution of object-level risk before critical events, while RiskWorld's selections preserve planning-critical information under filtered observation.
Introduction
RiskWorld frames driving risk as the future evolution of each candidate object's relation to the ego vehicle, rather than a static scene-level or post-hoc signal. It implements this framing with an object-centric latent world model that rolls relation-aware states forward and decodes object-level risk.
- Motivation: Risk identification must determine which surrounding participant or obstacle will become safety-critical to the ego vehicle before danger is visually obvious.Risk is object-specific and temporally evolving because current appearance or proximity may not reveal future path conflict.
- Motivation: Existing approaches estimate scene-level accidents, infer risky objects from ego reactions, or apply geometric checks after trajectory forecasting.These paradigms provide complementary signals but remain weakly coupled to direct object-level risk localization.
- Motivation: RiskWorld treats risk as the outcome of an evolving ego–object relation and predicts object-conditioned future relations before assigning object-level risk.Relevant relations include relative motion, distance, path conflict, and interaction trends toward unsafe states.
- Approach: RiskWorld combines pretrained predictive video representations, structured ego–object histories, relation-aware object states, RSSM-style latent dynamics, and risk-oriented decoders.The decoders produce object risk scores alongside auxiliary future-relation and temporal-risk predictions using only current and historical observations at inference.
- Results: RiskWorld improves overall F1 by 1.2 points over the strongest baseline on RiskBench and achieves the lowest false-alarm rate of 2.1%.Additional analyses examine temporal separation and planning relevance.
Related Work
Prior work spans scene-level accident anticipation, indirect behavior-based object localization, trajectory-based geometric checks, predictive video representations, and world models for future scene or latent-state prediction. RiskWorld builds on these lines while targeting object-level risk identification through relation-aware latent rollout.
- Risk identification: Accident anticipation methods primarily estimate scene-level collision or accident risk from dashcam video, with newer methods making future temporal targets more explicit.Examples predict multiple future horizons or classify predicted future latent representations, while object attention may remain explanatory rather than an object-risk target.
- Risk identification: Object-level risk identification asks which participant is the risk source, using ego-behavior inference or trajectory forecasting followed by distance or collision checks.RiskBench standardizes localization, anticipation, and planning-aware evaluation across scenario types.
- World models: World-model research rolls predictive states forward for occupancy, flow, point clouds, 4D scenes, controllable video, trajectory generation, or trajectory selection.RSSM-based latent dynamics and predictive video representations provide foundations for imagination-based future prediction without reconstructing future pixels.
Method: RiskWorld
RiskWorld encodes observed scene, object-track, and ego-motion information into relation-aware object states, then imagines each candidate’s future relative to the ego vehicle. Object-wise RSSM-style latent dynamics and auxiliary relation and temporal-risk decoders produce object-level risk evidence using only observed history at inference.
- Observation and formulation: RiskWorld estimates each candidate’s risk from an H-step latent rollout of future ego–object relations rather than the current frame alone.Logged future annotations supervise the rollout during training but are not inference inputs.
- Scene encoding: A frozen V-JEPA2 encoder extracts global and object-aligned predictive visual features from the observed clip, without reconstructing future pixels.Current boxes guide object-level feature pooling, while averaging spatiotemporal tokens gives a global scene feature.
- Object and relation encoding: Structured object encoding combines current ego-relative state and tracked history with ego motion, then relational fusion and masked attention contextualize ego–object and object–object interactions.The resulting relation-aware object tokens initialize the latent rollout.
- Object-wise latent future rollout: An RSSM-style deterministic–stochastic dynamics module evolves each relation-aware object token in compact latent space, initializing from the observed boundary and rolling forward with learned prior means.Dynamics parameters are shared across candidates, and no future observation is provided during the rollout.
- Risk-aware decoding and learning: Step-wise heads decode future ego-relative position, distance, and temporal-risk trajectories, while a pooled future state predicts the object risk score.Minimum distance and time-to-risk are derived from decoded trajectories; validity masks and weighted losses combine the supervised objectives.
Experiments
RiskWorld is evaluated on causal object-level risk localization, future-risk prediction, component contributions, and planning relevance across diverse RiskBench scenarios. It achieves the strongest overall localization balance, improves future-risk forecasting, and shows that predictive context, object grounding, and relational reasoning are important to performance.
- Risk localization: RiskWorld achieves the best overall F1 of 63.0 and the lowest non-interactive false-alarm rate of 2.1%.It improves overall F1 over the strongest baseline by 1.2 points while maintaining higher precision across interactive, collision, and obstacle subsets.
- Risk localization: RiskWorld suppresses nearby distractors while retaining risk evidence beyond fixed-range cutoffs.Recall is 76.7% below 5 m, 66.7% at 5–10 m, and 25.0% at 10–20 m, while corresponding false-positive rates below 10 m are 22.7% and 13.2%.
- Risk localization: Overall F1 remains stable around η = 0.70 while non-interactive false-alarm rate decreases as the threshold increases.The broad operating-point plateau permits recall–false-alarm adjustment without retraining.
- Future-risk prediction: RiskWorld reduces future-risk Brier error by 59.8% at 3 s compared with current-state persistence.Lower Brier scores hold across the full 3 s horizon, indicating more accurate tracking of evolving risk than static prediction.
- Future-risk prediction: Risk-source scores separate increasingly from distractors before critical events, with the source top-ranked in 95.9%–98.0% of scenarios at the critical frame.Source scores rise across interactive, collision, and obstacle scenes while the strongest non-risk score remains below 0.08.
- Ablation study: Removing world features reduces overall F1 from 63.0 to 50.6 and increases false-alarm rate from 2.1% to 21.3%.Removing object visual tokens or relation attention also lowers F1, showing complementary contributions from scene context, object grounding, and relational contextualization.
Discussion and Limitations
RiskWorld provides selective risk evidence beyond proximity, and its selected objects retain planning-relevant information under filtered observation. However, evaluation remains limited to simulation and does not directly demonstrate closed-loop planning improvements.
- Future ego–object relation rollout provides selective risk evidence beyond proximity.
- Filtered-observation evaluation suggests that selected objects retain planning-relevant information.
- RiskWorld remains a risk monitor rather than a closed-loop planner, so planning-aware evaluation does not measure direct planning improvements.
- Current evaluation is limited to simulation, with complex multi-object interactions, long-tail behaviors, and severe occlusions remaining open challenges.
Conclusion
RiskWorld constructs relation-aware object states from observed histories, rolls them forward with latent dynamics, and decodes future ego–object evolution into object-level risk evidence. On RiskBench, it achieves 63.0% overall F1 with a 2.1% false-alarm rate, while further analyses support early discrimination and planning relevance.
- RiskWorld constructs relation-aware object states from observed visual and motion histories and rolls them forward with RSSM-style latent dynamics.
- RiskWorld decodes imagined ego–object evolution into object-level risk scores and future-relation evidence.
- 63.0% overall F1 and a 2.1% false-alarm rate are achieved on RiskBench.
- Temporal and filtered-observation analyses support early risk-source discrimination and the planning relevance of RiskWorld’s selections.