PaperlayerView exact revision on arXiv

Source-linked AI summary

Risk-Aware Decision-Making for Autonomous Overtaking: A World Model-Based Mixture-of-Experts Framework

Yongzhi Liu, Sunan Zhang, Jinchang Xu, Jiawei Wang, Yushu Qiu, Chen Lv, Weichao Zhuang

arXiv:2609.00385v1cs.ROcs.AIcs.LG

TL;DR

Autonomous overtaking requires foresighted control because traffic interactions are stochastic and safety risks accumulate over multiple steps. WM-RMoE combines latent world-model rollouts, risk-aware expert gating, and GMM-based multimodal optimization. Experiments report stronger safety, robustness, and distinct maneuver generation across traffic densities.

  • Problem

    Highway overtaking lacks reliable explicit modeling of multi-step risk propagation, while unimodal policies can collapse distinct feasible maneuvers into unsafe averages.

  • Method

    WM-RMoE uses latent world-model rollouts for cumulative trajectory-risk assessment, hierarchical gating across three experts, and GMM-based optimization to preserve maneuver modes.

  • Results

    WM-RMoE significantly outperforms representative baselines in safety compliance, decision robustness, and generalization across varying traffic densities.

  • Takeaways & Limitations

    The framework supports foresighted, safety-compliant, and semantically distinct overtaking maneuvers while retaining adaptive switching between safety and efficiency policies.

Abstract

from arXiv · show

Autonomous highway overtaking demands foresighted decision-making to handle complex interactions, stochastic traffic evolution, and temporal risk accumulation. However, standard safe reinforcement learning approaches typically rely on implicit value-based risk estimations rather than explicit dynamics modeling, thereby struggling to accurately capture complex risk propagation over multi-step horizons. This limitation frequently results in behaviors that are locally safe but induce substantial latent risks in the long term. To address this, a World Model-based Risk-aware Mixture-of-Experts (WM-RMoE) framework is proposed. First, a learned latent dynamics model facilitates parallel multi-step rollouts, elevating safety assessment from the action level to the trajectory level via cumulative risk evaluation. Second, to enhance robustness under varying interaction intensities, a hierarchical gating mechanism dynamically coordinates experts across long-horizon, short-horizon, and rule-based safety modules. Furthermore, a Gaussian Mixture Model is integrated to preserve multimodal maneuvering branches, thereby mitigating the issue of behavioral mode averaging. Experimental results demonstrate that WM-RMoE significantly outperforms representative baselines in terms of safety compliance, decision stability, and generalization capability. Furthermore, benefiting from the risk-aware formulation, the proposed framework uniquely exhibits the ability to generate foresighted and semantically distinct overtaking maneuvers across diverse traffic densities.

I. INTRODUCTION

Highway overtaking requires decisions that anticipate stochastic, multi-step interactions and accumulating safety risks. WM-RMoE addresses this gap through trajectory-level risk assessment, risk-aware expert coordination, and multimodal planning.

  • Highway decisions must anticipate evolving multi-step risks while balancing safety and efficiency under stochastic traffic interactions.
  • Existing SafeRL methods commonly assess risk at the action level, limiting explicit modeling of how risk propagates across trajectories.
  • WM-RMoE uses world-model rollouts to estimate cumulative trajectory risk and remove hazardous candidates before execution.
  • A hierarchical risk-aware router coordinates heterogeneous learned, exploratory, and rule-based experts according to latent risk levels.
  • GMM-guided CEM preserves distinct maneuver branches and avoids the unimodal mode averaging that can produce infeasible mean actions.
  • Evaluations against SafeRL and MBRL baselines report superior safety compliance, task efficiency, and robustness under stochastic traffic dynamics.

A. CMDP Formulation for Highway Driving

The paper formulates highway control as a CMDP and learns an RSSM latent world model for long-horizon imagination, prediction, and trajectory-level safety evaluation.

  • A. CMDP Formulation for Highway Driving: The highway problem is modeled as a CMDP with continuous observations and actions, unknown transition dynamics, reward utility, safety costs, and a discounted cost budget.
  • A. CMDP Formulation for Highway Driving: The objective maximizes expected cumulative reward subject to a discounted safety-cost constraint.
  • B. RSSM Latent World Model: An RSSM maps high-dimensional observations into compact latent state sequences and learns temporal evolution for multi-step prediction.
  • B. RSSM Latent World Model: The world model predicts rewards and costs from latent states through dedicated reward and cost heads.
  • B. RSSM Latent World Model: Its variational objective combines reconstruction and prediction terms while regularizing the posterior toward the prior with KL divergence.
  • B. RSSM Latent World Model: The resulting latent predictive backbone supports trajectory-level risk evaluation and planning rather than instantaneous action assessment.
  • B. RSSM Latent World Model: After training, latent imagination recursively predicts future states, rewards, and costs for an action sequence over horizon H.

III. METHODOLOGY

WM-RMoE combines latent dynamics, heterogeneous policy experts, risk-aware gating, and multimodal trajectory optimization. Candidate sequences are rolled out and filtered by long-horizon rewards and safety costs before action selection.

  • The framework couples a latent world model, heterogeneous mixture-of-experts module, and risk-aware optimization pipeline.
  • The latent model encodes traffic observations, while the MoE router blends expert biases to generate diverse candidate action sequences.
  • Candidates undergo parallel latent rollouts, cumulative reward-cost evaluation, and GMM-based refinement of feasible solutions.
  • The long-term learned expert maximizes long-horizon return under expected safety-cost constraints using SAC-Lag.
  • The short-term memory expert propagates the preceding step’s multimodal solution space to preserve maneuver continuity and reduce search overhead.
  • The rule-based safety expert uses IDM for longitudinal control and MOBIL for lateral decisions, providing interpretable kinematic fallback behavior.
  • A dynamic gating mechanism fuses the experts’ complementary profiles to improve adaptability and safety margins in interactive scenarios.
  • Fig. 2 compares vanilla MoE with risk-aware MoE, whose router uses closed-loop uncertainty feedback to allocate sampling across experts.

2) Risk-Aware Expert Gating with Uncertainty Feedback:

The framework uses uncertainty-feedback gating to allocate sampling across heterogeneous experts, then optimizes multimodal candidate trajectories with world-model rollouts and trajectory-level risk filtering.

  • Risk-aware gating: The gating network combines the current latent state with prior elite-trajectory dispersion to adapt expert allocation to environmental context and planning reliability.High dispersion indicates weaker consensus among predictive plans and serves as a surrogate for planning risk.
  • Risk-aware gating: High uncertainty shifts probability toward the rule-based safety expert, while low uncertainty favors exploitation through the learned expert.The resulting ensemble uses latent-state inputs for learned and memory experts, but raw observations for the rule-based expert.
  • Risk-aware gating: The system contracts exploration in high-risk scenarios and expands it during safe navigation.
  • Trajectory optimization: The planner samples heterogeneous action sequences, rolls them through learned latent dynamics, and predicts long-horizon reward and cost trajectories.Parallel rollout makes future consequences of mixed expert strategies explicit before selection.
  • Trajectory optimization: Candidate trajectories are truncated at predicted terminal states, filtered by a cumulative discounted-cost budget, and ranked by expected cumulative return.Only feasible high-return trajectories form the elite set for subsequent refinement.
  • Trajectory optimization: The elite set is refit with a Gaussian mixture model, preserving distinct feasible maneuver branches and preventing mode collapse.The final distribution’s entropy also supplies planning uncertainty feedback for the gating loop.

D. Optimization Objectives and Training Procedure

The learned long-term expert is trained through differentiable latent dynamics and Lagrangian optimization, using imagined trajectories to balance reward maximization with cumulative safety costs.

  • Optimization objectives: The constrained problem is converted into an unconstrained dual objective, and the learned expert is optimized by analytic gradient propagation through latent dynamics.This design targets sample-efficient convergence to a safety-constrained policy.
  • Training procedure: Dual reward and cost critics estimate expected objectives from finite-horizon imagined trajectories using λ-return targets.These estimates provide global performance and risk signals for long-horizon policy optimization.
  • Optimization objectives: The actor minimizes the Lagrangian while the multiplier maximizes penalties for constraint violations, with gradients backpropagated through the learned transition function.
  • Training procedure: The resulting analytic optimization is designed to keep the learned expert compliant with trajectory-level safety constraints.

E. Risk-Guided Distillation of the Gating Network

The gating network is distilled from risk-constrained planner behavior, with uncertainty-modulated supervision that prioritizes the safety expert when planning ambiguity is high.

  • Distillation motivation: Because iterative GMM-CCEM planning is computationally intensive, the gating network is trained to imitate its expert-selection logic for real-time inference.
  • Target construction: The baseline supervision target is derived from empirical expert-selection statistics in the planner’s elite trajectories.
  • Risk-guided supervision: The rectified target mixes the empirical selection target with the safety expert according to planning uncertainty.The safety component saturates when uncertainty exceeds a risk threshold.
  • Distillation objective: An advantage-weighted cross-entropy objective emphasizes critical scenarios according to planned-trajectory value, while entropy regularization limits premature collapse.

IV. EXPERIMENTAL SETUP

Experiments use highway-env to evaluate constrained lane keeping and overtaking on a four-lane unidirectional highway with stochastic surrounding traffic and fully observable local kinematics.

  • Scenario: The evaluation platform is highway-env configured as a four-lane unidirectional highway, where the ego vehicle performs lane keeping and overtaking amid stochastic traffic.
  • Task formulation: The task is formulated as a Constrained Markov Decision Process balancing reward maximization against safety-cost constraints.
  • State space: The state concatenates ego kinematics with relative states of up to K = 4 nearest neighboring vehicles within the perception range.
  • State space: State features include positions, velocities, and headings; missing neighbors are zero-padded and all features are normalized to [−1, 1].
  • Action space: Actions control continuous longitudinal acceleration and steering angle, with tanh-squashed outputs mapped to physically feasible kinematic bounds.

B. Reward and Cost Function Formulation

The framework combines efficiency and comfort rewards with explicit safety costs for hazardous states, then evaluates real-time feasibility through processor-in-the-loop deployment.

  • B. Reward and Cost Function Formulation: The objective combines a dense reward function with a composite cost signal to promote compliant driving under safety constraints.The reward targets driving efficiency and passenger comfort, while the cost penalizes hazardous states.
  • B. Reward and Cost Function Formulation: The total reward aggregates efficiency and passenger comfort during highway driving.Efficiency is tied to normalized ego velocity, while comfort penalizes control variations.
  • B. Reward and Cost Function Formulation: The efficiency reward encourages high-speed cruising by normalizing ego velocity between the task limits vmin and vmax.This term relates the ego vehicle’s velocity to the permitted operating range.
  • B. Reward and Cost Function Formulation: The comfort reward penalizes jerk |∆a| and steering-rate variation |∆δf| to discourage abrupt control changes.Both longitudinal and lateral control smoothness are represented.
  • B. Reward and Cost Function Formulation: The safety cost penalizes hazardous states, including continuous environmental risk and discrete lane-violation or collision events.The Dynamic Risk Field integrates static road-boundary risks and dynamic surrounding-traffic risks; lane violations and collisions incur penalties of 5.0.
  • C. Implementation and PIL Validation Platform: Experiments use highway-env with 20-second episodes at 10 Hz, while IDM and MOBIL govern surrounding-vehicle behavior.The computational platform includes an Intel i7-14700 CPU, 32 GB RAM, and an NVIDIA RTX 4060 GPU.
  • C. Implementation and PIL Validation Platform: The Processor-in-the-Loop platform deploys the pretrained policy on an EAORA07 controller with dual NVIDIA Orin NX chips.A Linux host runs highway-env, and RTPS over Ethernet synchronizes observations and control commands.

D. Baselines and Evaluation Metrics

The evaluation compares WM-RMoE with rule-based, model-free, and model-based baselines using efficiency, robustness, and safety metrics. Across 300k training steps and five seeds, WM-RMoE shows faster and more stable convergence.

  • D. Baselines and Evaluation Metrics: The comparison spans rule-based, model-free, and model-based approaches for comprehensive evaluation.The baselines include IDM-MOBIL, SAC, SAC-Lag, CVPO, DreamerV3, and SafeDreamer.
  • D. Baselines and Evaluation Metrics: SafeDreamer extends DreamerV3 by integrating cost constraints into latent planning.DreamerV3 provides the model-based world-model baseline, while SafeDreamer adds safety awareness.
  • D. Baselines and Evaluation Metrics: The evaluation uses five principal metrics covering driving efficiency, algorithmic robustness, and safety adherence.Average Episode Reward and Average Speed measure task efficiency and traffic-flow capability.
  • D. Baselines and Evaluation Metrics: Average Travel Distance measures mean longitudinal distance, while Success Rate measures the ratio of collision-free episodes.Average Episode Cost quantifies the magnitude of constraint violations.
  • D. Baselines and Evaluation Metrics: All algorithms are trained for 3 × 10^5 steps using five random seeds to assess sample efficiency and convergence stability.Comparative learning curves are reported in Fig. 6 with shaded standard-deviation regions.
  • D. Baselines and Evaluation Metrics: WM-RMoE converges faster and stabilizes at the highest asymptotic reward, while SAC and CVPO learn more slowly with larger variances.The method also rapidly converges toward minimal cost and maintains strong success-rate performance.

B. Robustness Evaluation under Varying Traffic Densities

Robustness is evaluated across low, medium, and high traffic densities, alongside planning-horizon and expert-component ablations. WM-RMoE retains strong safety and efficiency under congestion, while removing experts or routing degrades targeted capabilities.

  • B. Robustness Evaluation under Varying Traffic Densities: Post-training robustness evaluation uses low, medium, and high densities defined relative to nominal density ρ0, with 100 trials per condition.The high-density regime is ρ = 1.5ρ0, while medium density is ρ = ρ0 ± 10%.
  • B. Robustness Evaluation under Varying Traffic Densities: WM-RMoE combines 100% SR with superior efficiency in low-density scenarios, outperforming the conservative SafeDreamer baseline.The framework performs online receding-horizon planning with latent trajectory imagination and dynamic expert routing.
  • B. Robustness Evaluation under Varying Traffic Densities: At ρ = 1.5ρ0, WM-RMoE achieves 91.6% SR, compared with 78.5% for SafeDreamer, while reaching average reward 152.3 and cumulative cost 17.8.These results are reported for the highly congested regime.
  • B. Robustness Evaluation under Varying Traffic Densities: Planning-horizon ablations compare H ∈ {5, 10, 15, 20} to examine the trade-off between foresight and stability.The reported figure identifies H = 15 as the optimal trade-off.
  • C. Ablation Analysis of Key Mechanisms: Removing the rule-based expert increases average speed to 32.3 m/s but raises cost to 21.1, indicating overly aggressive maneuvers.The rule-based expert functions as a kinematic safety bound.
  • C. Ablation Analysis of Key Mechanisms: Removing the GMM expert lowers success rate to 74.5% and increases instability by weakening preservation of multimodal action hypotheses.The resulting planner becomes susceptible to hazardous mode collapse.
  • C. Ablation Analysis of Key Mechanisms: Removing the RL expert lowers AER to 148.6, while replacing dynamic routing with uniform weights degrades all performance metrics.The ablations associate the RL expert with efficiency and the router with context-aware scheduling.
  • C. Ablation Analysis of Key Mechanisms: Training shifts expert utilization from rule-based and GMM safety priors toward the RL expert as learned value functions become more influential.The reported evolution links early safe priors with accelerated convergence and later RL authority with efficiency optimization.

D. Qualitative Analysis of Adaptive Decision-Making

Qualitative analyses show adaptive expert allocation, multimodal trajectory preservation, and counterfactual risk filtering within latent rollouts. Together, these mechanisms support trajectory-level mitigation while retaining distinct overtaking behaviors.

  • D. Qualitative Analysis of Adaptive Decision-Making: During a lane-changing episode, expert authority shifts across time as the vehicle approaches slower traffic and interaction intensity changes.The long-term learned policy dominates initially, followed by progressive activation of the short-term memory module.
  • D. Qualitative Analysis of Adaptive Decision-Making: GMM-CCEM samples many H-horizon candidates in parallel and preserves distinct action branches in the steering-acceleration plane.The analysis compares initial priors with elite subsets across t ∈ {1, 15, 30, 45}.
  • D. Qualitative Analysis of Adaptive Decision-Making: Standard CEM averages disjoint modes into a hazardous mean, such as accelerating without adequate steering.This unimodal collapse can produce a kinematically infeasible action.
  • D. Qualitative Analysis of Adaptive Decision-Making: GMM-CCEM maintains mode separation and tracks the optimal maneuver center as the overtaking sequence advances.Gaussian confidence ellipses represent component covariance structure, while purple shift vectors show maneuver-center movement.
  • D. Qualitative Analysis of Adaptive Decision-Making: The world model evaluates thousands of hypothetical trajectories and filters them using expected costs during receding-horizon planning.This provides a latent-space comparison between predicted evolution and ground-truth environmental evolution.
  • D. Qualitative Analysis of Adaptive Decision-Making: The Unsafe rollout demonstrates counterfactual risk reasoning by predicting a hypothetical lateral collision and enabling preemptive pruning of that sequence.The collision-prone trajectory is marked in red in the rollout visualization.
  • VI. CONCLUSION: WM-RMoE combines latent-space foresight with heterogeneous experts to shift from myopic single-step constraint satisfaction to farsighted trajectory-level risk mitigation.The conclusion also reports improved collision avoidance, decision robustness, multimodal preservation, and adaptive switching between safety priors and efficiency optimization.
  • VI. CONCLUSION: Future work targets centralized parallel rollouts, personalized driver profiles, and deployment on physical autonomous platforms.These directions address computational fidelity, behavioral diversity, and real-world kinematic consistency.
Loading 2609.00385v1…