PaperlayerView exact revision on arXiv

Source-linked AI summary

SRL-MPC: Shape-Aware Reinforcement Learned Model Predictive Control

Ruihua Han, Rui Gao, Zhe Liu, Xinyi Wang, Chang Chen, Shuai Wang, Qi Hao, Jia Pan, Hengshuang Zhao

arXiv:2608.21175v1cs.ROcs.AI

TL;DR

Dense-crowd navigation with heterogeneous shapes remains difficult because existing methods often simplify geometry, assume restricted settings, or struggle to generalize and scale. SRL-MPC combines GSF-based HOCBF constraints with RL-driven MPC parameter adaptation while retaining explicit MPC control computation. In randomized held-out crowd scenarios, it achieves the highest success rates among representative baselines, particularly at higher densities, with measured per-robot controller times remaining at the 10 ms level in tested dense scenarios.

  • Problem

    Safe real-time navigation among dense, dynamic, arbitrarily shaped objects remains challenging because existing approaches often rely on simplified geometry, restricted robot or workspace assumptions, or limited generalization.

  • Method

    SRL-MPC uses support-function-derived GSFs to formulate HOCBF safety constraints, while RL adapts explicit HOCBF-MPC parameters from neighboring geometries.

  • Results

    SRL-MPC achieves the highest success rate among representative baselines, reaching 92.0% with 25 robots and outperforming rule_mpc by 17.0 percentage points at that density.

  • Takeaways & Limitations

    The method preserves explicit shape-aware MPC safety structure while improving robustness and adaptability across randomized crowd densities and shapes.

  • Takeaways & Limitations

    The evaluation focuses on randomized convex polygon scenarios with held-out seeds; full perception pipelines, heterogeneous dynamics, nonconvex decomposed objects, and stronger centralized baselines remain future work.

Abstract

from arXiv · show

Safe and efficient shape-aware navigation in heterogeneous crowds and robot fleets remains challenging. Traditional approaches often assume homogeneous robots, sparse workspaces, simplified geometry, offline computation, or handcrafted parameters to make the problem tractable, which limits their deployment in dense crowd scenarios. Toward this end, we propose Shape-Aware Reinforcement Learned Model Predictive Control (SRL-MPC), a method for safe, efficient, and adaptive navigation in crowds with heterogeneous shapes without geometry simplification. To encode shape-aware safety, we formulate high-order control barrier function (HOCBF) constraints from geometric separation features (GSFs) based on support function transformation. A reinforcement learning (RL) framework then learns a neural policy that reads GSFs and outputs real-time MPC parameter updates, enabling the MPC solver to adapt to neighboring crowd geometries. The key advantage of SRL-MPC is that it preserves the safety structure and generalizability of MPC while integrating the adaptability and intelligence of RL. Experiments in randomized crowd scenarios with arbitrary shaped robot fleets demonstrate the effectiveness, scalability, and robustness of SRL-MPC. The results show that SRL-MPC substantially outperforms representative baselines in safety and adaptability. Project website: https://hanruihua.github.io/srl_mpc_project/

1 Introduction

Safe navigation among dense crowds is difficult when robots have heterogeneous, arbitrary shapes and must satisfy safety, efficiency, and kinematic constraints in real time. SRL-MPC addresses this by combining explicit shape-aware HOCBF-MPC safety constraints with RL-based online parameter adaptation, outperforming representative baselines in randomized dense-crowd experiments.

  • Existing navigation methods often assume homogeneous robots, sparse workspaces, simplified shapes, offline computation, or repeatedly tuned parameters.These assumptions limit deployment in dense and dynamic crowds, where robots may get stuck or collide without explicit geometric information.
  • Dense, dynamic crowds with arbitrarily shaped objects challenge RL generalization and large-crowd scalability, while learned costs or dynamics may weaken explicit shape-aware safety.Prior hybrid methods mainly target single-robot agile control rather than multi-robot shape-aware navigation.
  • SRL-MPC formulates geometric separation features as HOCBF constraints and uses RL to adjust MPC parameters from neighboring-agent geometry.The local MPC retains an explicit shape-aware safety problem while RL adapts its parameters online.
  • SRL-MPC outperforms representative baselines in task completion, safety, and robustness, especially as crowd density increases.Evaluation uses randomized multi-robot scenarios with random positions, goals, and shapes.

2 Related Work

Prior work spans geometric optimization, reinforcement learning, and hybrid methods for crowd navigation. SRL-MPC differs by adapting explicit shape-aware HOCBF-MPC parameters from local geometric features while retaining MPC as the executed controller.

  • Traditional Approaches: Optimization-based methods improve geometric fidelity using disk primitives, polytopic velocity obstacles, sequential convex optimization, and convex-set collision constraints.Exact constraints can grow with the number of object surfaces; SRL-MPC instead uses fixed-dimensional GSFs derived from support functions.
  • Reinforcement Learning Approaches: RL navigation methods learn policies from interaction data, local observations, social rewards, attention, and richer crowd representations.The related methods include socially aware and crowd-aware policy-learning approaches.
  • Hybrid Methods: Hybrid methods combine learning adaptability with model-based collision avoidance through learned policies, MPC execution, safety layers, or learned dynamics and costs.Examples include RL-RVO, social-navigation MPC coupling, DR-MPC, and learning-based MPC variants.
  • Hybrid Methods: Unlike prior hybrids, SRL-MPC uses RL to adapt HOCBF-MPC parameters from local GSFs while computing executed controls through explicit shape-aware MPC optimization.It neither learns the entire navigation policy nor uses optimization only as a safety shield.

3 Problem Statement

The problem statement defines local receding-horizon navigation for a robot among dynamic and static neighbors, balancing path tracking and control effort while enforcing shape-aware separation. Direct minimum-distance constraints are coupled and nonconvex, motivating later geometry updates and local optimization.

  • Each robot plans state and control sequences over an MPC horizon for a local crowd containing robots, pedestrians, and static or dynamic obstacles.The feasible set includes kinematics, input bounds, input-increment bounds, and the initial condition.
  • The navigation objective follows a reference path while maintaining a desired control profile through weighted position, heading, and control-effort terms.The cost balances position tracking, heading tracking, and control effort.
  • Collision avoidance is defined by the minimum Euclidean distance between the robot’s occupied set and each neighboring occupied set.The required safety margin is denoted by dsafe.
  • The local planning problem enforces Dij,k ≥ dsafe across neighbors and prediction steps alongside kinematic feasibility.The decision variables are the planned state and control sequences.
  • The coupled shape-aware planning problem is nonconvex and difficult to solve directly, motivating separation of geometry updates from local motion optimization.The next formulation adapts key MPC parameters online.

4 Method

SRL-MPC combines shape-aware HOCBF constraints derived from geometric separation features with reinforcement-learned MPC parameter adaptation. The framework decomposes geometry updates from local optimization and uses neighboring geometry to adapt navigation behavior.

  • Shape-Aware HOCBF Constraint: SRL-MPC formulates shape-aware safe sets as degree-2 HOCBF constraints from geometric separation features obtained through support-function transformation.The GSF includes a separating direction and minimum distance, while the HOCBF condition enforces safety over the prediction horizon.
  • Problem Decomposition: The GSF update subproblem fixes nominal trajectories and computes geometric separation features, while the local HOCBF-MPC subproblem fixes the separation direction and computes controls.This decomposition separates geometry updates from local motion optimization in the coupled bi-convex problem.
  • Local HOCBF-MPC: After decomposition, HOCBF violations enter local MPC through a soft residual penalty that is zero when safety is satisfied and positive when predicted trajectories violate it.The soft formulation is intended to avoid infeasibility from imposing high-order conditions as hard local constraints.
  • Online Execution: At each control cycle, nominal trajectories, GSFs, and local HOCBF-MPC solutions are updated iteratively before the first optimized control is applied.The reinforcement-learned adaptation can allow the decomposed subproblems to be solved with a single iteration, reducing computational cost.
  • Reinforcement Learned MPC: A reinforcement-learned policy maps neighboring GSFs and self-observation to updates of tracking, control-effort, and safety-distance MPC parameters.The policy outputs parameter increments, with admissible ranges for wp, wu, and dsafe, rather than directly producing control actions.
  • Observation and Policy: The compact observation combines GSF tensors for nearby neighbors with self-observation, leaving goal tracking and collision avoidance to HOCBF-MPC rather than an end-to-end policy.Neighbor features are stacked over the MPC horizon, zero-padded when fewer than K neighbors are active, and encoded for the policy network.

5 Experiments

SRL-MPC is evaluated in randomized polygonal-robot crowds against representative baselines, ablations, and perturbations. It maintains strong success and safety as density increases, while learned continuous adaptation improves over fixed or rule-based parameters.

  • Evaluation setup: Evaluation uses held-out random seeds and sweeps crowd size from 10 to 25 robots, with 100 episodes per robot count.The model is trained only in 15-robot scenarios and is tested on held-out episodes across densities.
  • Baseline comparison: SRL-MPC achieves the highest success rate across all robot counts, reaching 92.0% with 25 robots versus 7.0% for ORCA and 21.0% for SARL.The improvement over the strongest external baseline is 71.0 percentage points at 25 robots.
  • Ablation study: At 20 and 25 robots, SRL-MPC reaches 93.0% and 92.0% success, compared with manual_mpc at 75.0% and 68.0%.rule_mpc improves over manual_mpc but remains below SRL-MPC at both densities.
  • Ablation study: At 25 robots, SRL-MPC reduces timeout to 2.0% while keeping collision to 6.0%, whereas dist_mpc collision rises to 92.0%.The static distance-margin penalty is insufficient as the primary safety mechanism in dense polygonal crowds.
  • Robustness analysis: With N = 15, success remains 100.0% under 0.5 s parameter delay and 99.0% under one-step action delay.Success is 83.0% with Gaussian neighbor-position noise of σ = 0.05 m; larger noise or action delay increases collisions.

6 Conclusion

The paper presents SRL-MPC as a shape-aware MPC framework that combines explicit geometric safety constraints with learned parameter adaptation. Across randomized density sweeps, it achieves the highest success rates, particularly where baseline performance degrades.

  • Method: SRL-MPC represents geometric constraints with GSFs and support function transformation, then formulates degree-2 HOCBF constraints in the coupled optimization problem.Online planning decomposes into a GSF update and a local HOCBF-MPC subproblem with a softly penalized residual.
  • Method: Reinforcement learning adapts MPC parameters from neighboring shape-aware geometric features while executed control remains the solution of an explicit MPC optimization problem.This preserves an explicit optimization-based control structure while introducing learned adaptation.
  • Results: Density-sweep experiments show that SRL-MPC achieves the highest success rate among representative baselines, especially in dense scenes where baselines degrade sharply.Ablations further report that the soft HOCBF penalty is more reliable than a static distance-margin penalty and that learned adaptation improves robustness over fixed handcrafted settings.

A.1 Kinematic and Geometric Model

The model combines differential-drive kinematics with convex body-frame occupied sets transformed into the world frame. Support functions and conic representations provide the geometric ingredients for shape-aware separation constraints.

  • Kinematic Model: The differential-drive state is s_i,k = [x_i,k, y_i,k, θ_i,k]^⊤ with control u_i,k = [v_i,k, ω_i,k]^⊤.The discrete dynamics use sampling time Δt and map state-control pairs through f_i.
  • Kinematic Model: Feasible trajectories satisfy the discrete dynamics, input bounds, input-increment bounds, and the current initial condition.The feasible set F_i contains all state-control sequences satisfying these requirements.
  • Geometric Model: Each convex body is represented by conic inequalities C_i = {z ∈ R^2 | G_i z ⪯_{K_i} h_i} and transformed as Z_i(s_i,k) = {R_i,k z + p_i,k | z ∈ C_i}.The rotation matrix R_i,k depends on the predicted orientation θ_i,k.
  • Geometric Model: The support function σ_C(ξ) is the supremum of ξ^⊤z over the body, with closed forms for circles and polygons.For circles it is r∥ξ∥_2; for polygons it is the maximum vertex projection.
  • Geometric Model: The formulation imposes pairwise geometric constraints G_i z_i ⪯_{K_i} h_i and G_j z_j ⪯_{K_j} h_j for the occupied sets.These constraints support the body-frame distance program used for shape-aware separation.

A.2 Limitations

SRL-MPC’s main limitation is the computational cost of solving a local HOCBF-MPC problem at every control step. Its evaluation and deployment scope also rely on specific sensing, prediction, footprint, and scenario assumptions.

  • Computational Cost: 5.79, 6.73, 9.37, and 10.34 ms are the mean per-robot controller times for 10, 15, 20, and 25 robots, respectively.The measurements remain at the 10 ms level in the tested dense scenarios, but the controller is heavier than lightweight reactive baselines.
  • Scope and Assumptions: The evaluation focuses on randomized convex-polygon scenarios with held-out seeds rather than full perception pipelines or broader heterogeneous settings.Nonconvex decomposed objects and stronger centralized shape-aware baselines remain future work.
  • Scope and Assumptions: The method assumes sufficient onboard computation, reliable state estimates, short-horizon nominal neighbor predictions, and convex or convex-decomposed footprints.

A.3 Broad Impact

The work targets safer autonomous navigation in shared spaces such as warehouses, service-robot environments, and heterogeneous robot fleets. Additional challenging rollouts include varied geometric layouts sampled from held-out seeds.

  • Broad Impact: The framework targets safety and reliability for autonomous robots navigating shared spaces with other robots and humans.The stated application settings include warehouses, service-robot environments, and heterogeneous robot fleets.
  • Broad Impact: Keeping executed control inside explicit MPC while adapting interpretable parameters with RL may reduce risks associated with opaque end-to-end policies.
  • Broad Impact: Figure 7 samples cross geometry, nonconvex union, circular polygon, and through-traffic scenarios from held-out seeds.

A.4 Additional Challenging Scenarios

Additional evaluations cover varied geometric layouts and a close two-robot interaction. The learned policy updates MPC parameters online in response to neighboring geometry, while controller timing is profiled across robot counts.

  • Additional Challenging Scenarios: Figure 7 includes cross geometry, nonconvex union, circular polygon, and through-traffic layouts sampled from held-out seeds.The rollouts illustrate polygonal robots navigating different geometry layouts while adapting MPC parameters.
  • Reactive Parameter Adaptation: During a close interaction, the policy updates w_p, w_u, and d_safe online as neighboring geometry changes.The changes are driven by current geometric separation features rather than a fixed parameter schedule.
  • Implementation and Profiling: The implementation settings cover controller, scenario, RL, neural-network, evaluation, and baseline parameters.These settings are listed in Tables 4, 5, and 6.
  • Implementation and Profiling: Per-robot per-step controller time is profiled over three episodes per cell, up to 50 steps per episode, using seed 100.Each sample is one method decision-call duration divided by the number of robots.
Loading 2608.21175v1…