PaperlayerView exact revision on arXiv

Source-linked AI summary

RL-Guided PAC-NMPC for Probabilistically-Safe Perception-Based Navigation in Unknown Environments

Adam Polevoy, Dillon Capalongo, Katherine Tang, Mark Gonzales, Marin Kobilarov, Joseph Moore

arXiv:2609.39854v1cs.RO

TL;DR

Perception-based navigation in unknown environments must balance performance and safety, but RL policies may lack hard constraints outside their training distribution. This paper combines RL-trained probabilistic models with PAC-NMPC; simulation and hardware results report improved robustness and long-range performance over the compared approaches.

  • Problem

    RL navigation policies can perform poorly outside their training distribution and generally lack explicit hard-constraint enforcement, a concern for safety-critical navigation in unknown environments.

    Source: §I · p. 2

  • Method

    The method uses RL-trained actor-critic and sensor prediction models within PAC-NMPC, applying hard constraints for statistical safety and value-function improvement guarantees.

    Source: §I · p. 2, C. Uncertainty-Aware Value Function… · p. 8

  • Results

    The approach was more robust to model mismatch, sim-to-real transfer, and distribution shift than the RL actor alone, and achieved superior long-range performance than PAC-NMPC alone.

    Source: §IX · p. 18

  • Takeaways & Limitations

    Simulation and hardware demonstrations show real-time navigation through unknown environments using high-dimensional, nonlinear, underactuated, stochastic dynamics and local perception.

    Source: §IX · p. 18

Abstract

from arXiv · show

In this paper, we present an approach for combining stochastic nonlinear model predictive control (SNMPC) and reinforcement learning (RL) to enable probabilistically-safe perception-based navigation in unknown environments. Our method first uses RL to train probabilistic actor-critic and sensor prediction models. We then leverage these probabilistic models in a sampling-based SNMPC framework known as Probably Approximately Correct (PAC)-NMPC, which uses hard constraints to enforce finite-time statistical guarantees on the probability of collision and value function improvement. By ensuring that our finite-horizon SNMPC policies decrease the value function in expectation, we can approach the long-horizon performance of the RL approach while satisfying probabilistic safety constraints. Through simulation experiments, we show that our approach can improve the safety of perception-based RL navigation policies and scale to high dimensional systems with large sensor input spaces and complex nonlinear dynamics. We also demonstrate our approach through hardware experiments, showing improved performance for vision-based navigation with an agile fixed-wing aerial vehicle in unknown environments.

I. INTRODUCTION

The paper combines RL-trained models with PAC-NMPC to pursue long-range, perception-based navigation while enforcing probabilistic safety guarantees. Its related-work discussion contrasts this approach with safe RL, learned NMPC initialization, waypoint methods, and RL-MPC hybrids.

I. INTRODUCTION

  • PAC-NMPC uses RL-trained actor, value, and sensor prediction models to combine long-range planning with finite-time statistical safety guarantees.The approach also uses value-function improvement guarantees and avoids an inner NMPC optimization loop during RL training.Source: §I · pp. 2–3

A. Safe RL

  • Safe RL methods can lose constraint guarantees outside training distributions, while some control-based approaches assume known safety functions or state-only observations.The cited discussion contrasts these limitations with statistical guarantees and control-barrier or safety-layer approaches.Source: A. Safe RL · p. 3

B. Learned NMPC Warm-start

  • Learned NMPC warm starts span trajectory, active-set, and generative-model approaches, but few address local perception in unknown environments.The discussion includes transformer and diffusion-based methods and notes that few learned warm starts plan with local perception in unknown environments.Source: B. Learned NMPC Warm-start · pp. 3–4

C. NMPC with Learned Waypoints

  • Learned-waypoint approaches often separate high- and low-level planning, limiting reasoning when planning and perception are tightly coupled during dynamic maneuvers..Source: C. NMPC with Learned Waypoints · p. 4

D. NMPC & RL Hybrids

  • Prior RL-NMPC hybrids can improve long-horizon behavior or convergence, but the reviewed approaches generally do not provide safety guarantees.The paper positions Actor-Critic PAC-NMPC as handling stochastic dynamics and value functions, probabilistic safety guarantees, sizable perception inputs, and hardware evaluation.Source: D. NMPC & RL Hybrids · p. 4

E. Vision-based Agile Fixed-Wing Flight

The paper considers vision-based navigation through unknown obstacles with an agile fixed-wing vehicle and partial onboard perception. It presents simulation and hardware demonstrations of probabilistically safe navigation, within a finite-horizon safety framework.

E. Vision-based Agile Fixed-Wing Flight

  • Simulation and hardware experiments demonstrate probabilistically safe perception-based navigation and obstacle avoidance for agile fixed-wing flight.The evaluation uses an unknown obstacle field and onboard perception.Source: E. Vision-based Agile Fixed-Wing Flight · pp. 4–5

III. PROBLEM FORMULATION

  • The problem is to reach a goal through an unknown, unstructured environment while avoiding partially observed obstacles.The robot uses onboard perception, and the paper models its state evolution as stochastic dynamics.Source: §III · p. 5

IV. BACKGROUND

  • The background section reviews PAC-NMPC and actor-critic RL as the approach’s main components..Source: §IV · p. 5

A. PAC-NMPC

PAC-NMPC samples feedback policies and optimizes a surrogate distribution to bound expected cost and constraint-violation probability. Its guarantees apply locally to finite-horizon policies, conditional on current observations.

A. PAC-NMPC

  • PAC-NMPC optimizes a distribution over feedback policies rather than an open-loop control sequence, providing PAC bounds on expected cost and constraint-violation probability.The policy is constructed from a nominal input trajectory and time-varying feedback gains.Source: A. PAC-NMPC · p. 5
  • The optimization iteratively adjusts the mean and variance of a Gaussian surrogate distribution to minimize upper PAC bounds on cost and constraint violation.The bounds are estimated from finite policy samples.Source: A. PAC-NMPC · pp. 5–6
  • The trajectory constraint indicator is binary, so its expected value represents the probability of constraint violation..Source: A. PAC-NMPC · pp. 5–6
  • The guarantees are local to each finite-horizon policy and conditional on current observations; they do not establish infinite-horizon closed-loop safety..Source: A. PAC-NMPC · p. 6

B. Actor-Critic RL

The approach combines actor-critic RL with PAC-NMPC, using learned models to guide optimization. Its actor policy warm-starts the controller, while a learned sensor model predicts observations along sampled trajectories.

B. Actor-Critic RL

  • The actor approximates the optimal policy and the critic approximates the action-value function; both rely on robot observations when the full environment state is only partially observed.The value function can be obtained by composing the actor and critic.Source: B. Actor-Critic RL · p. 6

V. ACTOR-CRITIC PAC-NMPC (AC-PAC-NMPC)

  • The method trains an actor, critic, and sensor prediction model so actor and critic evaluations can use predicted future measurements along sampled trajectories.The learned actor and critic do not themselves provide safety guarantees; PAC-NMPC incorporates them to guide optimization.Source: §V · p. 6

A. RL-based Warm Start

  • The actor policy initializes the SNMPC distribution mean, with high variance for exploration; a final low-variance optimization can provide tighter guarantees for the executed policy.If the actor warm-start is severely infeasible, the method instead initializes from the previously optimized PAC-NMPC policy.Source: A. RL-based Warm Start · p. 7

B. Uncertainty-Aware RL-Augmented Cost

The learned value function extends finite-horizon NMPC toward longer-horizon perception-based cost-to-go, while probabilistic models support uncertainty-aware planning. A separate value-improvement constraint requires expected cost-to-go reduction over the planning horizon and provides a distinct probabilistic guarantee.

B. Uncertainty-Aware RL-Augmented Cost

  • The learned value function serves as a terminal cost to address NMPC’s finite-horizon myopia.The planning horizon may be limited by computational resources, especially for robots with complex dynamics and large state spaces.Source: B. Uncertainty-Aware RL-Augmented Cost · p. 7
  • Because the value function depends on sensor measurements, predicted terminal measurements let trajectory costs approximate perception-based cost-to-go.The paper contrasts this with, where the value function was not sensor-dependent.Source: B. Uncertainty-Aware RL-Augmented Cost · p. 7
  • Monte Carlo dropout models stochastic actor, critic, and sensor networks so PAC-NMPC can account for value-function and dynamics uncertainty at run time.Dropout masks are sampled across hidden layers; the learned value-function distribution is used for uncertainty-aware planning.Source: B. Uncertainty-Aware RL-Augmented Cost · p. 7
  • Sampling different dropout masks per trajectory adds no forward passes because PAC-NMPC already runs these networks for each sampled trajectory.The authors select MC dropout for its negligible training-time and inference-time cost.Source: B. Uncertainty-Aware RL-Augmented Cost · pp. 7–8
  • For simple 2D environments with large fields of view, learned sensor prediction may not be necessary for successful navigation.Source: B. Uncertainty-Aware RL-Augmented Cost · p. 8

C. Uncertainty-Aware Value Function Improvement Constraint

  • A value-improvement constraint requires the PAC bound on terminal value to be below its initial value, indicating expected cost-to-go reduction over the finite horizon.The value-improvement and constraint-violation bounds provide two distinct probabilistic guarantees, and both PAC bounds are hard optimization constraints.Source: C. Uncertainty-Aware Value Function… · p. 8

VI. RALLY CAR UGV WITH LIDAR: APPROACH AND EVALUATION

This section applies AC-PAC-NMPC to a 1/10th-scale LiDAR-equipped rally car and summarizes an existing formulation and experiment. The platform provides a comparison basis for a more complex system, particularly because it does not use the RL warm-start or learned perception prediction described earlier.

VI. RALLY CAR UGV WITH LIDAR: APPROACH AND EVALUATION

  • The rally-car platform provides a comparison basis for a more complex system, but its prior experiment does not use the RL warm-start or learned perception prediction.Source: §VI · p. 8

A. Rally Car Dynamics and LiDAR Sensor

  • The rally car is modeled with stochastic bicycle dynamics and equipped with a 64-beam, 360° planar LiDAR that returns range measurements at 64 bearings.The state includes position, orientation, forward speed, and steering angle; the input is forward acceleration and steering-angle rate.Source: A. Rally Car Dynamics and LiDAR Sensor · p. 8

B. Actor-Critic

  • Actor and critic networks are trained with TD3 using sampled environments and a constraint-violation penalty based on full environmental knowledge.The networks use normalized observations that include speed, steering, goal information, and LiDAR measurements.Source: B. Actor-Critic · pp. 8–9

C. Sensor Prediction

  • For this planar 2D system with a 360° field of view, LiDAR prediction uses geometric projection and does not quantify prediction uncertainty.Current valid returns are projected to obstacle points; predicted measurements are assigned to the nearest discrete sensor bearing.Source: C. Sensor Prediction · p. 9

D. Simulation Experiments

The simulations evaluate PAC-NMPC against multiple baselines in cluttered and concave-trap environments, while hardware tests examine transfer and navigation with learned models. The fixed-wing evaluation uses a stochastic dynamics model and depth-image sensing.

1) Cluttered Environments:

E. Hardware experiments

  • On the rally-car hardware, AC-PAC-NMPC outperformed the actor and never collided, including when its value function was trained with an incorrect 0.5 m wheelbase.Trajectory sampling used the correct wheelbase in the mismatch test.Source: E. Hardware experiments · p. 10, Table IV · p. 10

VII. FIXED-WING UAV WITH DEPTH CAMERA: APPROACH AND EVALUATION

  • For the fixed-wing vehicle, learned sensor prediction, RL-based warm-starting, and a separate value-function improvement bound were identified as critical to successful navigation.The evaluation used a 32-inch-wing-span vehicle with a depth camera and more complex dynamics and sensing than the rally car.Source: §VII · pp. 10–11

A. Fixed-wing Dynamics and Depth Camera Sensor

  • The model defines state using position, orientation, linear and angular velocity, and control-surface angles; controls specify surface angular rates and throttle.The nominal continuous-time dynamics also account for mass, inertia, thrust, gravity, aerodynamic-surface forces, and drag.Source: A. Fixed-wing Dynamics and Depth… · p. 11
  • The fixed-wing stochastic dynamics use second-order Runge-Kutta integration and Gaussian process noise fitted from hardware flight data.The sensor returns a flattened depth image with an 87° × 58° field of view.Source: A. Fixed-wing Dynamics and Depth… · p. 11
  • The fixed-wing tests used a learned dynamics model and depth camera, with model acceleration errors characterized using hardware flight data.The fitted Gaussian distribution is shown in Figure 9.Source: A. Fixed-wing Dynamics and Depth… · p. 11, Figure 9 · p. 11

B. Actor Critic Training

The actor and critic use normalized state and depth-image inputs, with the camera image reduced to 16 × 12 pixels for rapid inference.

B. Actor Critic Training

  • Actor and critic networks use two 256-neuron hidden layers, 10% dropout, and ReLU activation, with 16 × 12 depth images as input.The authors note that this low resolution may limit detection of thin obstacles, but was sufficient for the targeted environments with obstacles 0.2–0.4 m in radius.Source: B. Actor Critic Training · pp. 11–12
  • Network inputs also include goal-relative features, position, orientation, control-surface angles, and body-frame linear and angular velocities; all inputs are normalized.The depth image is flattened before it is included among the inputs.Source: B. Actor Critic Training · p. 12

C. Sensor Prediction

The sensor-prediction section motivates and evaluates learned future depth-image prediction for trajectory evaluation. The associated fixed-wing experiments compare controller variants, including the full approach, in concave-trap environments.

C. Sensor Prediction

  • Geometric projection produced inconsistent future depth predictions and left many pixels unassigned, so the authors trained a sensor-prediction model with supervised learning.Training examples pair the current state and image with a future state, targeting the image at that future time.Source: C. Sensor Prediction · p. 12
  • The U-Net outperformed the fully connected model on simulated depth-image prediction, using smooth L1 plus SSIM loss and evaluation on a 10% held-out validation set.Table V reports the model comparison.Source: C. Sensor Prediction · p. 12, Table V · p. 12

1) Controller Configuration:

  • The controller uses a history of up to 10 depth images when obstacles leave the limited camera field of view, and replans every 0.1 seconds.The authors report that the increased replanning rate improved reactivity to obscured obstacles given the vehicle speed and limited field of view.Source: 1) Controller Configuration · p. 13

2) Environments:

  • To address traps that challenge finite-horizon planning, the simulated environments included randomly placed concave obstacles, and depth images included obstacles, walls, ceiling, and floor.Trap count ranged from zero to six; obstacle radii ranged from 0.2 to 0.4 m.Source: 2) Environments · p. 13

3) Baseline Comparison:

  • The baseline sequence isolates mapping, global planning, and unknown-space penalties; the unknown-space variant assigns an extra A* cost of 100 to unobserved cells.The variants include voxel-grid mapping and a 3D A* planner with a receding-horizon goal.Source: 3) Baseline Comparison · pp. 13–14
  • AC-PAC-NMPC reached the goal without obstacle-constraint violations in 90% of trials, the highest success rate, and had the lowest successful-trial average accumulated cost among PAC-NMPC baselines.The authors attribute the successful, low-cost navigation to RL guidance, which also outperformed PAC-NMPC with mapping and a global planner.Source: Table VI · p. 13, 3) Baseline Comparison · p. 14

4) Ablation Study:

Hardware trials gave the full approach the highest success rate and lowest successful-trial cost among the compared methods. Ablations and out-of-distribution tests assess the contributions of learned sensor prediction, RL warm-starting, and robustness to obstacle changes.

4) Ablation Study:

  • Removing either RL-based warm-starting or learned sensor prediction left successful navigation in only a small percentage of trials.The authors suggest inaccurate geometric-projection predictions explain the sensor ablation, while critic estimates may be unreliable on trajectories far from the actor.Source: 4) Ablation Study · p. 14

5) Out of Distribution Experiment:

  • With a horizontal obstacle outside the actor and critic training distribution, AC-PAC-NMPC succeeded in 76% of trials versus 46% for the RL actor.The approach matched the best baseline's success rate and had lower average accumulated cost over successful trials.Source: 5) Out of Distribution Experiment · p. 14, Table VIII · p. 14

E. Hardware Experiments

  • In hardware, AC-PAC-NMPC had the highest success rate at 80% and the lowest average accumulated cost over successful trials; both baselines reached 40%.A spurious depth-camera noise episode caused a spike in the probability-of-constraint-violation bound.Source: E. Hardware Experiments · p. 15, Table IX · p. 15
  • Hardware flight paths demonstrate real-time control through an unknown environment using local perception while providing probabilistic safety guarantees.The hardware comparison found degraded performance for mapping-based PAC-NMPC, with intermittent sensor noise remaining in the map despite filtering.Source: E. Hardware Experiments · pp. 15–16, Figure 16

VIII. ANALYSIS

The analysis examines whether a penalty-based RL objective can support constraint satisfaction without sacrificing value-function descent. Under stated assumptions, a sufficiently large constraint penalty and an existing feasible policy yield a feasible nearby control that descends the value function.

A. One-Step Feasible Descent

  • With a sufficiently large constraint penalty and an existing feasible policy, a feasible control near the penalty-optimal action can still decrease the value function.The result concerns the existence of a nearby feasible input, not a guarantee that the optimal or learned policy is itself constraint-free.Source: §VIII · p. 16, A. One-Step Feasible Descent · p. 16

VIII. ANALYSIS

  • A constraint-penalized training cost does not guarantee that the resulting optimal or learned policy is constraint-free, especially outside the training distribution.The analysis assumes a non-negative constraint function and a positive penalty coefficient.Source: §VIII · p. 16

A. One-Step Feasible Descent

  • The proof first establishes a neighborhood of non-optimal actions with negative expected one-step value change, then uses continuity and a sufficiently large penalty to reach feasibility.The neighborhood argument relies on continuity of the value change around the optimal action.Source: A. One-Step Feasible Descent · p. 16

B. Multi-Step Feasible Descent

The one-step feasible-descent result is extended to feedback policies over multiple steps. Given the theorem's assumptions, a feasible policy parameterization near the optimal one can satisfy constraints throughout the horizon while descending the value function.

B. Multi-Step Feasible Descent

  • The multi-step setup parameterizes feedback actions by ξ and defines the resulting multi-step dynamics and feedback cost over the horizon.The feedback cost is built from the policy's costs across the horizon.Source: B. Multi-Step Feasible Descent · p. 17
  • Under the theorem's assumptions, a feasible multi-step feedback policy exists near the optimal policy and descends the value function while satisfying constraints at every step.The policy parameterization must be rich enough to represent the optimal multi-step feedback policy.Source: B. Multi-Step Feasible Descent · p. 17
  • The extension applies the one-step result to ξ using continuity of the multi-step quantities and an adequately expressive policy parameterization.The resulting feasible parameters satisfy the constraints across the horizon and yield negative value change.Source: B. Multi-Step Feasible Descent · p. 17

C. Approximate Value Functions

The paper reports that RL-guided PAC-NMPC navigates in simulation and hardware, with greater robustness than the RL actor and better long-range performance than PAC-NMPC alone. Its analysis also notes that the deployed learned value function is only an approximation whose gap from the optimal value is unknown.

C. Approximate Value Functions

  • The analysis assumes an optimal value function, but the deployed learned actor and critic are uncertain approximations; because the approximation gap is unknown, the method aims only for a decrease in the approximate value.The text notes that further analysis of the gap between estimated and true value functions remains future work.Source: C. Approximate Value Functions · p. 17, §IX · p. 18

D. Simulation Experiment

  • AC-PAC-NMPC successfully navigated at smaller γc values than the actor alone, and warm-starting with a prior policy when the actor trajectory was infeasible significantly improved safety.Figure 18 presents success and collision rates for each constraint-penalty value.Source: D. Simulation Experiment · p. 17, Figure 18 · p. 18

IX. CONCLUSION

  • In simulation and hardware, the approach navigated unknown environments and was more robust to model mismatch, sim-to-real transfer, and distribution shift than the RL actor, with superior long-range performance to PAC-NMPC alone.The conclusion identifies learned actor-critic and sensor-prediction models as components of the approach.Source: §IX · pp. 17–18
Loading 2609.39854v1…