Source-linked AI summary
Socially Aware Motion Planning with Deep Reinforcement Learning
Yu Fan Chen, Michael Everett, Miao Liu, Jonathan P. How
TL;DR
Socially compliant navigation is difficult to quantify because human behaviors and feature statistics are stochastic, limiting direct imitation approaches. The paper develops SA-CADRL, which uses deep reinforcement learning and norm-violation penalties to induce time-efficient social behaviors, and demonstrates fully autonomous navigation at human walking speed among pedestrians.
Problem
Human social-navigation mechanisms are difficult to quantify because people’s behaviors and feature statistics vary across individuals and runs, challenging feature-matching approaches.
Method
SA-CADRL uses deep reinforcement learning with norm-inducing penalties and a symmetrical neural network to learn socially aware multiagent collision avoidance.
Results
The robotic vehicle navigated fully autonomously at 1.2m/s in a dynamic indoor environment with many pedestrians while maintaining safe distances and generally respecting social norms.
Takeaways & Limitations
Social norms such as passing on the right and overtaking on the left can emerge from time-efficient reciprocal collision avoidance and be induced through simple behavioral biases.
Abstract
from arXiv · showhide
For robotic vehicles to navigate safely and efficiently in pedestrian-rich environments, it is important to model subtle human behaviors and navigation rules (e.g., passing on the right). However, while instinctive to humans, socially compliant navigation is still difficult to quantify due to the stochasticity in people's behaviors. Existing works are mostly focused on using feature-matching techniques to describe and imitate human paths, but often do not generalize well since the feature values can vary from person to person, and even run to run. This work notes that while it is challenging to directly specify the details of what to do (precise mechanisms of human navigation), it is straightforward to specify what not to do (violations of social norms). Specifically, using deep reinforcement learning, this work develops a time-efficient navigation policy that respects common social norms. The proposed method is shown to enable fully autonomous navigation of a robotic vehicle moving at human walking speed in an environment with many pedestrians.
I. INTRODUCTION
Pedestrian-rich navigation requires robots to move safely and efficiently despite subtle, stochastic social behaviors that are difficult to quantify. The paper introduces SA-CADRL, a deep-RL approach that induces social norms and demonstrates autonomous robotic navigation at human walking speed.
- Motivation: Pedestrian-rich environments require efficient, safe navigation near people whose subtle social norms and intents are difficult to quantify.Robots must account for how their motion affects nearby pedestrians.
- Existing approaches: Reactive obstacle-avoidance methods can generate unsafe or unnatural movements near human walking speed because they do not capture human behavior.
- Existing approaches: Learning-based methods more closely resemble human behavior but rely on stochastic feature statistics that vary across people and runs, raising generalization concerns.
- Proposed direction: Rather than precisely modeling social-compliance mechanisms, the paper uses simple social norms such as passing on the right within a reinforcement-learning framework.
- Contributions: SA-CADRL introduces socially aware collision avoidance, generalizes to multiagent scenarios with a symmetrical neural network, and demonstrates autonomous navigation at human walking speed.
II. BACKGROUND
The background formulates multiagent collision avoidance as sequential decision-making under uncertainty about nearby agents’ hidden intents. Deep reinforcement learning addresses the resulting continuous, high-dimensional value-function problem.
- Problem formulation: Each agent’s state includes observable position, velocity, and size alongside unobservable goal, preferred speed, and orientation, while its action is velocity.
- Problem formulation: The policy minimizes expected time to goal while avoiding collisions with nearby agents.
- Problem formulation: The expectation accounts for nearby agents’ unobservable intents and policies, making their motion uncertain.
- Reinforcement-learning formulation: A reward function rewards reaching the goal and penalizes collisions, while the unknown transition model captures uncertainty in neighboring agents’ motion.
- Deep reinforcement learning: Deep neural networks represent value functions in the continuous, high-dimensional joint state space, avoiding impractical discretization and enumeration.
B. Characterization of Social Norms
Social norms are difficult to specify procedurally because human motion is stochastic, yet CADRL can exhibit conventions through reciprocal, time-efficient collision avoidance. However, its emergent convention can conflict with human expectations and shift when efficiency favors another path.
- Motivation: Human navigation norms such as passing on the right are intuitive but difficult to quantify because motion varies across individuals.Variation includes speed and smoothness, and the difficulty increases with more nearby pedestrians.
- Emergent norms: Reciprocal, time-efficient local interactions can produce complex normative motion without explicitly specifying procedural rules.Reciprocity models other agents’ behavior and supports cooperation without communication, while allowing either left- or right-handed conventions.
- CADRL behavior: CADRL agents prefer passing on the right in a symmetric case, and this preference remains robust under a small initial offset.The policy exhibits the convention without imposed behavioral rules.
- CADRL behavior: With a larger initial offset, CADRL changes passing direction to favor shorter, smoother paths.The behavior balances the navigation convention against time and path efficiency.
- Limitations: CADRL’s emergent convention can conflict with human interpretation because agents with different sizes and preferred speeds prefer passing on the left.The behavior also depends on state features and can vary with value-network initialization and training cases.
III. APPROACH
SA-CADRL first shapes normative behavior for two agents within a deep reinforcement learning framework, then generalizes the method to multiagent scenarios.
- III. APPROACH: SA-CADRL shapes normative behaviors for a two-agent system using deep reinforcement learning before generalizing to multiagent scenarios.The approach is presented as socially aware multiagent collision avoidance with deep reinforcement learning.
A. Inducing Social Norms
SA-CADRL induces social norms by penalizing configurations that violate desired passing, overtaking, and crossing conventions. A local goal-aligned state representation and norm-specific penalty regions guide reinforcement learning without precisely specifying compliant trajectories.
- The agent state uses a local coordinate frame aligned with its goal to reduce redundancy from rotation and translation.
- The state includes goal distance, relative-agent position and velocity, relative heading, radius, and a real-or-virtual-agent indicator.The other agent’s distance and heading are explicitly defined in the state parameterization.
- Norm violations are specified as penalty regions, allowing desired behaviors to be induced without precisely defining compliant motion procedures.The added penalty breaks symmetry in collision avoidance and favors behaviors respecting the selected norm.
- The right-handed reward penalizes overtaking, passing, and crossing configurations identified by the sets S_ovtk, S_pass, and S_cross.The corresponding regions are illustrated in Fig. 4, where they map to shaded areas around the red agent.
- 700 episodes were sufficient for convergence with the stated penalty sets, whereas a 30% smaller set required 1250 episodes.Larger penalty sets could cause instability or divergence, while converged policies were largely insensitive to penalty-set size.
B. Training a Multiagent Value Network
The multiagent value network directly handles scenarios with more than two agents using symmetry-preserving architecture and norm-inducing rewards. Training generates trajectories on random cases, converts them into value experiences, and supports both left- and right-handed policies.
- The method addresses higher-order multiagent interactions by training directly on multiagent scenarios rather than scaling a two-agent network with minimax.
- Weight sharing and max pooling make the network invariant to permutations of nearby agents while preserving the self-agent distinction.The architecture uses two symmetric layers, one max-pooling layer, and two fully connected layers.
- The n-agent input concatenates the agent state with the observable states of n−1 nearby agents, and the total reward combines collision and norm-inducing rewards.The norm penalty is applied when the closest nearby agent places the joint configuration inside S_norm.
- Training initializes a value network from an n-agent trajectory dataset, generates random-case trajectories with an ϵ-greedy policy, and assimilates state-value pairs into experience sets.
- Trajectory reflection across the x-axis explores left- and right-handed path topologies more efficiently, avoiding wrong passing sides after 2000 episodes without the procedure.Virtual agents allow an n-agent network to generate trajectories for scenarios with fewer agents.
A. Computational Details
The multiagent implementation is tuned for real-time action queries and evaluated on pedestrian-like agent sizes and speeds. Four-agent training is substantially more computationally expensive than two-agent training, while repeated runs converge to similar norm-respecting policies.
- 8.7ms is the average value-network query time for a four-agent SA-CADRL policy on an i7-5820K CPU.The query produces an action in the Python implementation.
- Nine hours were required for 3,000 four-agent training episodes, compared with two hours for 1,000 two-agent episodes.The four-agent system took longer because its state space is higher-dimensional.
- Repeated training runs converged to similar policies that exhibited the respective desired social norms on all evaluation-set test cases.
- Random test cases used pedestrian-like radii r ∈ [0.2, 0.5]m and preferred speeds v_pref ∈ [0.3, 1.8]m/s.The collision reward specifies a desired minimum separation of 0.2m.
B. Simulation Results
Simulation tests show that norm-inducing rewards make SA-CADRL strongly prefer the specified passing convention while retaining time-efficient navigation, with a measurable cost in travel time.
- The left- and right-handed policies always selected the respective correct passing side on perfectly symmetrical test cases.
- On randomly generated two-agent cases, all reinforcement-learning methods produced more time-efficient paths than ORCA.
- The four-agent SA-CADRL policies trained with norm rewards showed an 85-15 preference split for their respective social norms.The norm-free policy developed conventions that did not consistently match human norms, including overtaking on the right.
- In four-agent tests, all reinforcement-learning methods outperformed ORCA, while SA-CADRL maintained larger average separation distances than CADRL.CADRL sometimes came within 0.1m of another agent because it was trained on a two-agent system and focused on the closest neighbor.
C. Hardware Experiment
The hardware system combines onboard perception, global planning, and SA-CADRL local avoidance to navigate autonomously among pedestrians at human walking speed. Across ten runs in daily indoor traffic, it maintained safe distances and generally followed right-handed conventions without safety-driver intervention.
- The robotic vehicle combines Lidar, three Intel RealSense cameras, and four webcams for localization, free-space detection, and pedestrian detection.Pedestrian tracking combines Lidar point clouds with camera images.
- Global planning uses diffusion maps to find a subgoal within a 5m horizon, while SA-CADRL selects local velocity actions among feasible directions.The local policy can respond to pedestrians detected beyond the static planning horizon.
- The complete sense-plan-execute cycle runs in real time at 10Hz on the onboard computer.
- Across ten runs, the vehicle navigated fully autonomously at 1.2m/s while an average of 10.2 pedestrians came within 2m each minute.The average distance between successive randomly generated goals exceeded 50m.
- The vehicle maintained safe pedestrian distances, generally respected social norms, and required no safety-driver intervention during the ten runs.Using right-handed norms, it generally passed pedestrians on the right and overtook on the left.
V. CONCLUSION
The paper presents SA-CADRL as a socially compliant multiagent collision-avoidance method that uses reinforcement learning and extends to more than two agents. A hardware implementation enabled autonomous navigation at human walking speed among many pedestrians.
- SA-CADRL learns socially compliant collision avoidance through reinforcement learning, including passing on the right and overtaking on the left in a right-handed system.
- A symmetrical neural network structure generalizes the approach to multiagent scenarios with n > 2.
- The robotic implementation enabled fully autonomous navigation at human walking speed in a dynamic environment with many pedestrians.Future work will consider relationships among nearby pedestrians, such as groups walking together.