Source-linked AI summary
Decentralized Structural-RNN for Robot Crowd Navigation with Deep Reinforcement Learning
Shuijing Liu, Peixin Chang, Weihang Liang, Neeloy Chakraborty, Katherine Driggs-Campbell
TL;DR
Robot crowd navigation must operate despite partial observability, dense crowds, and uncertain human dynamics. The paper introduces DS-RNN, which combines spatio-temporal graph reasoning with model-free deep reinforcement learning without expert supervision. It reports better performance than prior methods in challenging scenarios and transfers the learned policy from simulation to a TurtleBot 2i.
Problem
Robot crowd navigation is difficult because robots have partial observations in dense crowds, while prior methods assume agent dynamics are known and well-defined.
Method
DS-RNN represents robot-crowd interactions as a decentralized spatio-temporal graph and learns a navigation policy with model-free deep reinforcement learning without expert supervision.
Results
The model outperforms previous methods in challenging crowd-navigation scenarios and transfers its learned policy from simulation to a real-world TurtleBot 2i.
Takeaways & Limitations
Combining spatial and temporal reasoning with model-free reinforcement learning provides a navigation policy reported to adapt to dense crowds and partial observability.
Takeaways & Limitations
The simulation keeps human behavior unaffected by the robot, real-world human detections are noisy, and each model uses a fixed number of humans.
Abstract
from arXiv · showhide
Safe and efficient navigation through human crowds is an essential capability for mobile robots. Previous work on robot crowd navigation assumes that the dynamics of all agents are known and well-defined. In addition, the performance of previous methods deteriorates in partially observable environments and environments with dense crowds. To tackle these problems, we propose decentralized structural-Recurrent Neural Network (DS-RNN), a novel network that reasons about spatial and temporal relationships for robot decision making in crowd navigation. We train our network with model-free deep reinforcement learning without any expert supervision. We demonstrate that our model outperforms previous methods in challenging crowd navigation scenarios. We successfully transfer the policy learned in the simulator to a real-world TurtleBot 2i. For more information, please visit the project website at https://sites.google.com/view/crowdnav-ds-rnn/home.
I. INTRODUCTION
Robot crowd navigation is challenging because decentralized robots operate with partial observability and must handle complex interactions in dense crowds. The paper proposes DS-RNN, a spatio-temporal, model-free reinforcement-learning framework that avoids assumptions about agent dynamics and expert policies.
- Partial observability makes other agents’ walking styles and intended goals difficult to infer online.
- Dense crowds combine dynamic and static agents whose interactions are difficult to model, producing challenging navigation behavior.
- Existing reaction-based and trajectory-based methods can freeze in dense crowds when planners judge all paths unsafe.
- Deep V-Learning methods typically rely on ORCA supervision and assume human dynamics are known and deterministic.
- DS-RNN models crowd navigation as a decentralized spatio-temporal graph and trains an end-to-end policy with model-free reinforcement learning.
- The proposed policy learns without expert supervision and is reported to outperform previous methods in dense crowds and partially observable scenarios.
A. Reaction-based methods
Prior crowd-navigation approaches use local interaction rules, trajectory prediction, or value learning, but face freezing, computational, modeling, and sensing limitations. The paper addresses these issues with model-free reinforcement learning and spatio-temporal recurrent reasoning.
- Reaction-based methods such as RVO, ORCA, and Social Force use current states or interaction rules to generate collision-free actions.
- These methods can produce shortsighted or unnatural paths and suffer from the freezing robot problem.
- RNNs process trajectory sequences while spatial and temporal graph structure organizes robot-crowd interactions into smaller factors.
- Trajectory-based methods enable long-sighted planning but can be computationally expensive because prediction and online path search operate over large state spaces.
- Deep V-Learning assumes known human state transitions, inherits demonstration-policy disadvantages, and requires increasingly difficult value estimation or sensing resources.
- The proposed model-free RL policy avoids state-transition models, Monte-Carlo value estimation, and expert supervision.
III. METHODOLOGY
The paper formulates crowd navigation as an MDP in which a robot acts from partially observed human and robot states, with unknown transitions and rewards accumulated over an episode. It then represents the problem using state, action, transition, reward, discount, and initial-state components.
- Problem formulation: Crowd navigation is modeled as an MDP where the robot interacts with humans through unknown state transitions.The process ends at a goal, collision, or maximum episode length.
- Feature construction: Spatial features describe human–robot relations, while temporal features describe the robot’s trajectory across time.These features provide the inputs used to represent the navigation state.
- State representation: The robot state includes position, velocity, goal, maximum speed, heading, and radius, while human observations omit velocity and radius.The omission reflects measurement difficulty in real-world deployment.
- MDP dynamics: At each timestep, the robot selects an action from its policy, receives a reward, and transitions to the next state.Human agents simultaneously follow their own policies and unknown transition probabilities.
- Objective: The discounted return sums future rewards, and the value function is the expected return from a state under policy π.The robot’s objective is to maximize expected return.
B. Spatio-Temporal Graph Representation
The crowd is represented as a decentralized spatio-temporal graph whose nodes are agents, spatial edges connect agents at one timestep, and temporal edges connect the same agent across adjacent timesteps. A factor-graph transformation decomposes the robot policy into learnable node and edge factors.
- Spatio-Temporal Graph Representation: The decentralized st-graph contains agent nodes, same-timestep spatial edges, and adjacent-timestep temporal edges.Edges and nodes with little influence on robot decisions are pruned.
- Factor Graph: The factor graph decomposes the robot policy into a robot node factor, spatial edge factors, and a robot temporal edge factor.These factors collectively determine the robot’s action from node and edge features.
- Parameter Sharing: All spatial edges share one factor, keeping the parameter count constant as the number of humans increases.Parameter sharing supports scalability with larger crowds.
C. Network Architecture
DS-RNN derives an end-to-end network from the factor-graph representation, using separate recurrent modules for spatial interactions, temporal dynamics, and robot-node processing. Attention aggregates spatial-edge information before the network produces the policy and value estimates.
- Network Architecture: DS-RNN uses spatial edgeRNN, temporal edgeRNN, and nodeRNN modules corresponding to the factor-graph components.The architecture is derived from the st-graph factorization and uses trainable weights and fully connected layers.
- Spatial Processing: The spatial edgeRNN processes each human–robot spatial feature and shares parameters across all human–robot pairs.Shared parameters allow the same module to process every spatial edge.
- Temporal Processing: The temporal edgeRNN captures the dynamics of the robot’s own trajectory over time.It linearly transforms temporal edge features before recurrent processing.
- Attention: An attention module assigns weights to spatial edges and combines their recurrent outputs into a weighted representation.The attention mechanism is similar to scaled dot-product attention.
- Decision Output: The nodeRNN combines robot state, attended spatial information, and temporal hidden states to produce the policy and state value.A fully connected layer maps the nodeRNN hidden state to V(s_t) and π(a_t|s_t).
- Training: The end-to-end DS-RNN performs spatial and temporal reasoning using three RNNs that learn factor parameters.The network is trained with PPO using model-free policy-gradient learning.
IV. SIMULATION EXPERIMENTS
The simulation environment places agents in a 12m × 12m 2D plane and visualizes robot, human, goal, orientation, and field-of-view elements. Humans inside and outside the robot’s field of view are distinguished visually.
- Simulation Experiments: The section presents the simulation environment used for training and simulation experiments.The environment illustration is adapted from prior work.
- Simulation Environment: The simulation uses a 12m × 12m 2D plane with circular humans, a yellow robot, and a red-star goal.Red arrows indicate agent orientations.
- Simulation Environment: Dashed lines mark the robot’s field-of-view boundaries, with blue humans inside and red humans outside.The color encoding distinguishes visible from out-of-view humans.
A. Simulation environment
The simulator uses holonomic agents whose actions specify desired x- and y-axis velocities, with immediate velocity execution over each timestep.
- Each agent uses holonomic kinematics with action a_t = [v_x, v_y].Agents immediately achieve desired velocities and maintain them for Δt seconds.
- Agent positions are updated from their current positions and velocity components over Δt.
- Humans follow ORCA with randomized maximum speeds and radii, while reacting only to other humans.The robot is excluded from human reactions to prevent learning an aggressively coercive policy.
1) Environment configurations:
The study evaluates navigation under limited field of view and dense mixed crowds, adding randomized goals and safety-oriented rewards to increase scenario complexity.
- Environment configurations:: The FoV environment varies the robot’s field of view from 0° to 360° while maintaining five humans per episode.Humans begin and end at random positions on a circle of radius 6m.
- Environment configurations:: The Group environment fixes a 360° field of view while varying dense crowds containing static circle groups and freely moving humans.It tests whether policies exhibit the freezing robot problem.
- Environment configurations:: Episodes randomize the robot’s initial position and goal, and humans may change goals during an episode.
- Environment configurations:: Humans immediately receive new random goals after reaching destinations, simulating continuous human flow.
- Environment configurations:: The reward combines goal-reaching rewards, collision and proximity penalties, and potential-based shaping toward the goal.The robot receives high reward when approaching the goal while maintaining safe human separation.
B. Experiment setup
The experiments compare DS-RNN with representative reaction-based, trajectory-based, and Deep V-Learning methods, including an RNN-based ablation.
- Experiment setup: ORCA and SF represent reaction-based methods, while RGL represents trajectory-based methods and Deep V-Learning.
- Experiment setup: CADRL and OM-SARL serve as additional Deep V-Learning baselines.
- Experiment setup: RNN+Attn isolates the effect of DS-RNN’s architecture by adding an RNN to OM-SARL with attention-weighted human features.Its RNN matches DS-RNN’s robot-node sequence length and size.
2) Training:
The evaluation trains comparable models and tests them across partial observability and crowd density, while trajectory comparisons examine navigation behavior.
- Training:: DS-RNN and RNN+Attn are trained for 1×10^7 timesteps with learning rate 4×10^-5, using the same reward and matched architectures across methods.
- Training:: Six experiments vary field of view at 90°, 180°, or 360° and crowd size at 10, 15, or 20 humans.Each experiment uses 500 random unseen test cases and measures success, collision, timeout, and successful-episode navigation time.
- Evaluation: DS-RNN has higher success and lower collision and timeout rates than RNN+Attn in all tested settings.It also has shorter navigation time because it often finds a better path.
- Spatio-temporal reasoning:: DS-RNN uses spatio-temporal reasoning to combine relevant edge and node features, enabling explicit reasoning about human relationships and robot dynamics.
- Evaluation: In Group environments, ORCA and SF show high timeout rates that increase with the number of humans, indicating prevalent freezing behavior.Their large navigation times also indicate overly conservative behavior in dense crowds.
- Comparison with traditional methods:: RNN-based policies use trajectory sequences for longer-sighted decisions, whereas ORCA and SF consider only the current state.
3) Comparison with Deep V-Learning:
Model-free reinforcement learning enables RNN+Attn and DS-RNN to outperform Deep V-Learning despite unknown agent transitions, while DS-RNN jointly learns interactions and decision making end-to-end. The policy was also evaluated on a TurtleBot 2i, though simulation assumptions, sensor noise, and fixed crowd sizes limit the results.
- Comparison with Deep V-Learning: Despite unknown state transitions, RNN+Attn and DS-RNN perform better in all metrics than Deep V-Learning methods.CADRL, OM-SARL, and RGL use value networks initialized by ORCA through supervised learning, whereas the model-free methods learn from scratch.
- Comparison with Deep V-Learning: RGL is competitive in some cases through spatial reasoning and human trajectory prediction, but its relational graph and planner remain separate modules.DS-RNN instead jointly learns robot-human interactions and decision making end-to-end.
- Real-world evaluation: The trained model was evaluated on a TurtleBot 2i mobile platform using depth-based human perception and robot localization.Human positions were obtained with an Intel RealSense D435, YOLOv3, and Deep SORT; an Intel RealSense T265 localized the robot.
- Limitations and conclusion: The study reports better performance in challenging simulation environments and promising real-world results, while identifying several deployment limitations.These include simulation without mutual robot-human effects, noisy real-world human positions, and fixed human counts that challenge generalization to real human flows.