PaperlayerView exact revision on arXiv

Source-linked AI summary

A Risk-Sensitive and Uncertainty-Aware Decision-Making and Control Framework for Safe and Robust Autonomous Driving

Zhuoren Li, Ran Yu, Weiqi Zhang, Ming Liu, Lu Xiong, Chen Sun, Bo Leng

arXiv:2609.09650v1cs.RO

TL;DR

Urban unsignalized intersections challenge RL because policies must handle interactive, safety-critical, OOD, and long-tail conditions without excessive conservatism. RUDC combines risk-sensitive distributional RL, ensemble policy-uncertainty quantification, and adaptive HOCBF correction; simulations report a balance of safety, efficiency, and robustness, including SR above 86% and VR under 10% in dense scenarios.

  • Problem

    Interactive unsignalized intersections produce stochastic safety-critical events, while standard RL can overlook tail risks and fixed safety filters can become excessively conservative.

  • Method

    RUDC couples risk-sensitive distributional RL with ensemble-based policy uncertainty quantification and uncertainty-aware HOCBF correction with residual compensation.

  • Results

    RUDC maintains SR above 86% while suppressing VR below 10% in dense scenarios, demonstrating a balance between safety-critical requirements and operational flexibility.

  • Takeaways & Limitations

    Adaptive uncertainty-based intervention supports trial-and-error near safe-set boundaries while avoiding the conservative deadlock of rigid safety margins.

Abstract

from arXiv · show

Reinforcement learning (RL) has demonstrated considerable potential for autonomous driving decision-making. However, its deployment in urban autonomous driving, particularly at highly interactive unsignalized intersections, remains challenging, as learned policies may struggle to maintain both safety and robust decision-making in complex traffic situations. Conventional safety-filtering approaches typically employ fixed conservative constraints, which may improve safety at the cost of excessive intervention and degraded traffic efficiency. To address these limitations, we propose a Risk-sensitive and Uncertainty-aware Decision-making and Control (RUDC) framework for safe and robust autonomous driving. RUDC couples risk-sensitive distributional RL with ensemble-based policy uncertainty quantification, jointly accounting for tail risks in return distributions and uncertainty in learned policies. An uncertainty-aware high-order control barrier function (HOCBF)-based safety correction mechanism adaptively adjusts constraint strictness according to policy uncertainty, while a learnable residual predictor compensates for CBF model mismatches and discretization errors. Extensive simulations at unsignalized intersections demonstrate that RUDC achieves a favorable balance among safety, efficiency, and robustness, outperforming representative safe RL baselines under both nominal and challenging OOD and long-tail scenarios while satisfying real-time requirements.

I. INTRODUCTION

Urban unsignalized intersections expose RL policies to interactive, stochastic, and safety-critical conditions where average-return optimization and fixed safety interventions may be insufficient. RUDC combines tail-risk modeling, policy-uncertainty quantification, and adaptive safety correction to balance reliability, safety, and efficiency.

  • Unsignalized intersections create complex multi-agent interactions and stochastic safety-critical events that increase OOD and long-tail risks.
  • RL policies lack inherent safety guarantees and reliable policy assessment, while conventional safe RL can face weak safety signals near feasible boundaries.
  • Expectation-based RL may favor high-average-return behaviors that remain vulnerable to rare but catastrophic outcomes.
  • RUDC couples risk-sensitive distributional RL with ensemble-based policy uncertainty quantification to account jointly for return-distribution tail risks and learned-policy uncertainty.
  • Its uncertainty-aware HOCBF correction adapts safety-constraint strictness to policy uncertainty and uses a residual predictor to compensate for CBF mismatches and discretization errors.
  • Benchmarking and OOD analyses evaluate RUDC against Safe RL baselines, including how uncertainty changes safety interventions in challenging traffic scenarios.

B. Control Barrier Functions

The paper frames safe control as maintaining forward invariance of a designated safe set while addressing the tail risks and uncertainty of interactive driving. It uses risk-sensitive distributional RL and uncertainty-aware HOCBF concepts to regulate nominal actions without uniformly rigid intervention.

  • Control Barrier Functions: A control-affine system is kept safe by maintaining its state within a forward-invariant set.
  • Control Barrier Functions: Discrete-time CBFs constrain state transitions through h(x_k) and action-dependent changes while preserving forward invariance under the stated class-K conditions.
  • Risk-sensitive Reinforcement Learning: Standard RL optimizes expected return, which can overlook rare catastrophic outcomes in interactive driving with asymmetric or multimodal return distributions.
  • Risk-sensitive Reinforcement Learning: Risk-sensitive distributional RL maximizes a distorted expectation of the return distribution, with CVaR used to formalize aversion to worst-case outcomes.
  • Framework Architecture: The framework architecture combines ensemble distributional critics, uncertainty decomposition, and uncertainty-embedded HOCBFs to adjust safety correction according to quantified reliability.

2) Ensemble Learning and Uncertainty Quantification:

The framework uses ensemble critics with bootstrapping and randomized prior functions to quantify uncertainty in risk-sensitive return distributions. It separates aleatoric fluctuation from epistemic disagreement, including disagreement in lower-tail expectations.

  • Ensemble construction: Deep ensembles estimate policy reliability by combining multiple critics designed to agree on in-distribution samples and remain diverse on OOD data.Bootstrapping gives ensemble members different replay-buffer subsets, while randomized prior functions help prevent prediction collapse in unfamiliar states.
  • Ensemble construction: Each ensemble member combines a trainable critic with a frozen randomized prior function to improve posterior estimates and preserve prediction diversity.The return uses the original network and fixed prior network, scaled by a prior factor.
  • Risk-sensitive learning: The actor maximizes a risk-averse return aggregated across ensemble critics using distributional quantile estimates and CVaR.Training uses quantile Huber loss, bootstrap masks, replay-buffer mini-batches, and ensemble target critics.
  • Uncertainty quantification: Total uncertainty is decomposed into aleatoric uncertainty from intrinsic tail fluctuations and epistemic uncertainty from disagreement among ensemble members.The decomposition follows the law of total variance and is extended to lower-β tail expectations for extreme-risk decisions.

1) Safety Correction Formulation:

The safety correction layer converts learned actions into minimally invasive safe controls using high-order control barrier functions. It models vehicle and road-boundary constraints with a control-affine single-track system and solves a quadratic program at each sampling instant.

  • HOCBF formulation: Because inter-vehicle distance constraints often have relative degree 2, standard CBFs do not expose acceleration controls in the first barrier derivative.High-order CBFs address this by recursively defining auxiliary functions and superlevel sets.
  • HOCBF formulation: The HOCBF construction recursively combines barrier derivatives with class K functions to enforce forward-invariant safety sets.The valid HOCBF condition is defined over a sequence of recursively constructed auxiliary functions and superlevel sets.
  • Vehicle and spatial modeling: The vehicle is represented by a single-track control-affine model with state [px, py, ψ, v]⊤ and control [alon, zf]⊤, where zf = tan δf.Relative kinematics include surrounding obstacles assumed to maintain constant velocity over the prediction horizon.
  • Vehicle and spatial modeling: Elliptical support radii and safety margins define candidate barriers for dynamic vehicles and static road boundaries.Road-boundary constraints project the EV center of gravity and wheelbase-based reference points onto lane edges.
  • Safety filtering: At each sampling instant, a quadratic program computes the minimally invasive safe action from the nominal RL action, then recovers the steering angle as δ⋆f = arctan(z⋆).The control weight matrix Wu = diag(wa, wz) weights the correction objective.

2) Uncertainty-embedded Constraints:

The uncertainty-embedded constraint adapts safety-filter strictness to policy uncertainty while correcting model mismatch with a learned residual predictor. Low uncertainty permits relaxation, whereas high uncertainty tightens the safety margin, especially in the long tail.

  • Adaptive safety constraints: The framework assumes conservative maneuvers under high uncertainty and greater flexibility under low uncertainty rather than fixed conservatism.This design targets a balance between safety and traffic efficiency.
  • Model uncertainty: A nonlinear control-affine model separates known nominal dynamics from unmodeled dynamic interactions embedded into the CBF safety condition.The uncertainty term is projected onto the barrier gradient as w(x, u) = Lφh(x).
  • Residual compensation: An MLP predicts one-step CBF residuals from normalized RL actions, ego state, and obstacle state using separate vehicle and road-constraint heads.The residual captures model mismatch between the predicted and true next-step barrier values.
  • Adaptive safety constraints: Negative predicted residuals tighten constraints because the nominal model overestimates safety, while low policy uncertainty permits relaxation.The adaptive slack δ(σk) is piecewise continuous and strictly decreasing with uncertainty.
  • Adaptive safety constraints: The 95th and 97.5th uncertainty percentiles anchor the slack schedule, with the tightening slope doubling beyond the higher threshold.The modulation gain η controls the uncertainty-dependent adjustment, and positive slack below the lower threshold promotes efficiency.

IV. IMPLEMENTATION

The evaluation uses a bidirectional four-lane unsignalized intersection in Highway-Env, with surrounding vehicles controlled by an improved IDM. Training mixes task distributions to support a generalizable policy, and testing is divided into distinct evaluation regimes.

  • Scenario and evaluation: The experiment models a bidirectional four-lane intersection in Highway-Env with surrounding vehicles controlled by an improved Intelligent Driver Model.The IDM predicts each surrounding vehicle’s heading and position over the subsequent 2 s while yielding according to road priorities.
  • Scenario and evaluation: The ego vehicle trains on a mixed task distribution, while testing is partitioned into two regimes with distinct evaluation settings.The mixed training distribution is intended to build a generalizable policy.

B. MDP Formulation

The MDP represents the ego vehicle, surrounding traffic, waypoints, and task objectives, with continuous acceleration and steering actions. Its reward combines sparse task outcomes with dense tracking, smoothness, progress, and safety terms, while the network processes variable intersection traffic for policy and value estimation.

  • 1) Observation and Action Spaces:: The observation combines ego state, surrounding-vehicle states, reference waypoints, and a one-hot maneuver task encoding.The task encoding covers left turn, going straight, and right turn.
  • 1) Observation and Action Spaces:: Surrounding vehicles are represented relative to the ego vehicle using position, velocity, and heading differences.Waypoints are also expressed as relative position offsets.
  • 1) Observation and Action Spaces:: The continuous action space consists of longitudinal acceleration and front-wheel steering angle.
  • 2) Reward Design:: The total reward combines sparse collision-completion terms with dense reference-line tracking, action smoothness, destination progress, and safety terms.The sparse component penalizes collisions and rewards successful task completion.
  • 2) Reward Design:: The dense reward also penalizes large or abruptly changing controls, supporting smoother behavior and energy efficiency.Reference tracking selects the smaller error among two candidate reference lines.
  • C. Network Architecture and Training Details: The architecture encodes heterogeneous state components, fuses waypoint and interaction features, and uses parallel decoders for policy distributions and quantile-based Z-value estimates.The design accommodates varying numbers and configurations of surrounding vehicles at unsignalized intersections.
  • C. Network Architecture and Training Details: Hyper-parameters and evaluation metrics for the decision and control algorithm are reported in Tables I and II.

D. Baselines and Evaluation Metrics

RUDC is evaluated against multiple safe RL baselines and ablations across nominal, dense, OOD, and long-tail intersection conditions. The results examine learning efficiency, safety violations, success, failure, and the trade-off between conservative filtering and adaptive exploration.

  • Baseline comparison: RUDC-E and RUDC-T rapidly converge to high episode rewards while maintaining the lowest violation rates among the compared baselines.The uncertainty-embedded HOCBF supports safe exploration without hindering learning progress.
  • Baseline comparison: In the Random Destination task, RUDC-T and RUDC-E achieve the highest SR of 92.3% and 91.7%, respectively.USL reaches 88.5% SR, while vanilla SAC has a 21.1% VR and Recovery RL has a 42.0% FR.
  • Challenging conditions: In the Dense Unprotected Left Turn task, RUDC-E and RUDC-T sustain SRs of 87.1% and 86.3% under elevated OOD and long-tail-event likelihood.The framework combines CVaR optimization of worst-case returns with HOCBF formal safety boundaries.
  • Ablation studies: Policies without explicit safety filters become overly aggressive, reaching a VR of 15.5% despite higher average speeds.The ablation comparison identifies ESAC and RUDC w/o CBF as lacking HOCBF safety filters.
  • Ablation studies: Only-TTCBF and Only-ECBF exhibit conservative deadlock, with FR exceeding 78% and SR below 13% because uncertainty-aware constraint adaptation is absent.Their low initial violation rates reflect near-stationary behavior that limits diverse interactive data collection.
  • Ablation studies: As training progresses, RUDC variants maintain SR above 86% while suppressing VR below 10% in dense scenarios.Uncertainty-embedded boundaries permit trial-and-error near safe-set boundaries instead of rigid conservative margins.

C. Case Studies in OOD Scenarios

The OOD case studies examine uncertainty-aware safety intervention, hyperparameter sensitivity, computational cost, and remaining CBF limitations. Together, these passages describe how RUDC responds to unexpected obstacles while balancing safety, efficiency, and real-time operation.

  • OOD scenario response: A sharp rise in σtotal near accident vehicles triggers stronger HOCBF correction, prompting slower steering that ultimately reaches the destination safely.The intervention expands the safety margin while avoiding obstacles and road boundaries.
  • Computational efficiency: Peak latency below 60 ms and average total latency of 21.88 ms satisfy the stated real-time requirement of at least 10 Hz.HOCBF optimization and critic inference account for approximately 43.8% and 40.7% of the average computational load, respectively.
  • Risk sensitivity: Lower CVaR β generally reduces violations, but extreme risk aversion can freeze the agent and degrade navigation efficiency.The analysis varies β from 0.1, described as highly risk-averse, to 1.0, described as risk-neutral.
  • Ensemble-size sensitivity: Increasing Nens. from 3 to 5 improves success rate and episode reward while reducing violation rate, whereas larger ensembles add cost without consistent gains.The study selects Nens. = 5 as the default because further increases produce marginal or fluctuating improvements and near-linear growth in parameter size and inference time.
  • Limitations and future work: The CBF formulation uses instantaneous state-action pairs without historical context, which can produce corrected actions that deviate substantially from the nominal policy.Future work proposes Predictive CBFs to integrate historical information and make interventions smoother and more consistent.

B. Truncated Taylor Control Barrier Functions

TTCBF approximates discrete-time CBF evolution with a truncated Taylor series, allowing safety constraints for relative-degree systems to be constructed using a class K function. For relative degree two, the formulation uses a linear class K function and a tuned parameter.

  • Taylor approximation: TTCBF approximates the discrete-time change in the barrier function with a truncated Taylor expansion through the system’s relative degree.The r-th derivative contains the control input, and the approximation includes terms through order r.
  • TTCBF definition: A TTCBF is defined using a class K function α(x) constrained by α(x) ≤ x over the relevant safe set.This formulation provides the condition used to construct the discrete-time barrier constraint.
  • Relative-degree-two construction: For relative degree two, TTCBF applies a linear class K function with λT1 ∈ (0, 1] to construct the constraint.The discrete sampling period Δt is part of the resulting constraint formulation.
  • Parameter selection: The implementation evaluates λT1 over [0.05, 1.0] and selects 0.1 for the reported experiments.This parameter is chosen analogously to the reported ECBF parameter-selection procedure.

APPENDIX B VALIDATION OF CBF RESIDUAL PREDICTOR

The appendix validates the residual-prediction component used to compensate HOCBF model mismatches and discretization errors, while also situating it within the paper’s constrained-RL safety formulation. The reported regression results show close agreement between predicted and true barrier variations.

  • Predictor role: The residual predictor estimates safety-boundary residuals needed to compensate HOCBF model mismatches and discretization errors.This residual is presented as important for the uncertainty-embedded HOCBF mechanism.
  • Validation results: R2 scores of at least 0.98 and tight alignment with the y = x line indicate accurate prediction of barrier variations for TTCBF and ECBF.The scatter plots compare predicted barrier variations against true observed values.
  • Validation protocol: The MLP predictor is evaluated on 105 random exploration samples using R2, MAE, and RMSE.The evaluation measures regression accuracy for both TTCBF and ECBF residual predictions.
  • Safety-RL context: The safe-RL formulation treats safety as a constrained optimization problem over reward and cumulative violation cost.The appendix describes CMDP and baseline mechanisms including Lagrangian relaxation, action projection, recovery policies, and critic-based cost estimation.
  • Cost formulation: The implementation combines continuous interactive risks with discrete event penalties in a unified step-wise cost function.Distance-based risk is normalized for vehicle and road channels before applying the stated safety thresholds and weights.
Loading 2609.09650v1…