PaperlayerView exact revision on arXiv

Source-linked AI summary

DRL-VO: Learning to Navigate Through Crowded Dynamic Scenes Using Velocity Obstacles

Zhanteng Xie, Philip Dames

arXiv:2301.06512v2cs.RO

TL;DR

Dense crowds and partial observability make fast, safe robot navigation difficult to design with manually specified rules or supervised data alone. The paper introduces a deep reinforcement learning policy combining lidar history, pedestrian kinematics, a sub-goal, and a velocity-obstacle reward, and reports stronger speed-safety balance and generalization across simulated and real-world settings.

  • Problem

    Dense crowds with moving pedestrians and static obstacles make it difficult to design clear model-based rules or collect sufficiently representative supervised-learning data for autonomous navigation.

  • Method

    The policy combines local-frame lidar history, pedestrian kinematics, and a sub-goal with a velocity-obstacle reward that guides active collision avoidance and goal progress.

  • Results

    The policy achieves a better collision-avoidance and speed balance, generalizes across crowd sizes and unseen environments, and works in tested real-world environments without retraining.

  • Takeaways & Limitations

    The reported results support using structured observations and velocity-obstacle guidance to transfer a crowd-navigation policy from simulation to varied environments and robot platforms.

  • Takeaways & Limitations

    The policy can struggle to find collision-free directions in very high crowd densities and with sudden changes in pedestrian motion.

Abstract

from arXiv · show

This paper proposes a novel learning-based control policy with strong generalizability to new environments that enables a mobile robot to navigate autonomously through spaces filled with both static obstacles and dense crowds of pedestrians. The policy uses a unique combination of input data to generate the desired steering angle and forward velocity: a short history of lidar data, kinematic data about nearby pedestrians, and a sub-goal point. The policy is trained in a reinforcement learning setting using a reward function that contains a novel term based on velocity obstacles to guide the robot to actively avoid pedestrians and move towards the goal. Through a series of 3D simulated experiments with up to 55 pedestrians, this control policy is able to achieve a better balance between collision avoidance and speed (i.e., higher success rate and faster average speed) than state-of-the-art model-based and learning-based policies, and it also generalizes better to different crowd sizes and unseen environments. An extensive series of hardware experiments demonstrate the ability of this policy to directly work in different real-world environments with different crowd sizes with zero retraining. Furthermore, a series of simulated and hardware experiments show that the control policy also works in highly constrained static environments on a different robot platform without any additional training. Lastly, several important lessons that can be applied to other robot learning systems are summarized. Multimedia demonstrations are available at https://www.youtube.com/watch?v=KneELRT8GzU&list=PLouWbAcP4zIvPgaARrV223lf2eiSR-eSS.

I. INTRODUCTION

The paper addresses autonomous navigation through dense crowds and static obstacles by combining deep reinforcement learning with structured sensor representations and a velocity-obstacle reward. The resulting policy is designed to balance collision avoidance with speed and generalize across crowd sizes and unseen environments.

  • Motivation: The paper targets navigation applications such as delivery through partially known spaces containing moving people and static obstacles.These applications require autonomous and fast movement to destinations.
  • Evaluation: The policy achieves a better balance between collision avoidance and speed and generalizes better than model-based, supervised-learning, and DRL baselines.The comparison covers different crowd sizes and unseen environments.
  • Motivation: Dense crowds make manually designed model-based rules and representative supervised-learning datasets difficult to obtain.The paper therefore uses deep reinforcement learning for environments containing dozens of moving pedestrians.
  • Approach: The velocity-obstacle reward penalizes headings likely to cause collisions while encouraging progress toward the goal.This provides a more active collision-avoidance signal than a sparse collision penalty.
  • Approach: The policy combines a short lidar history, nearby-pedestrian kinematics, and a sub-goal point in the robot’s local coordinate frame.These representations encode static obstacles, dynamic-obstacle motion, and destination information while improving robustness to localization errors.

III. NAVIGATION POLICY

The navigation policy treats robot control as a partially observable decision problem: partial observations from limited-FOV sensors are transformed into features, then an actor produces steering actions while a critic estimates state value.

  • Problem Formulation: Limited sensor fields of view make complete environmental perception unavailable in real-world navigation.The robot therefore constructs a partial observation from sensor data before selecting an action.
  • Problem Formulation: The navigation problem is formulated as a partially observable Markov decision process with state, action, transition, reward, observation, and observation-probability components.Deep reinforcement learning optimizes a neural policy in this formulation.
  • Network Architecture: The network uses feature generation, feature extraction, actor, and critic modules to convert partial observations into control actions and state values.The feature extractor produces high-level features, while the actor and critic operate separately.
  • Observation Space: The observation consists of lidar history, pedestrian kinematics, and sub-goal position, all expressed in the robot’s local reference frame.A consistent 80 × 80 grid format is used for the input types.

1) Pedestrian Kinematics:

The policy preprocesses lidar and goal information into compact representations, using pooled short-history lidar data and a sub-goal selected along the nominal path.

  • Lidar History: The lidar input uses 0.5 seconds of data, or 10 scans, with minimum and average pooling stacked into an 80 × 80 array.The lidar is sampled at 20 Hz.
  • Lidar History: Minimum-plus-average pooling yielded the best supervised validation performance among the tested lidar preprocessing operations.Performance was measured by MSE between learned and expert velocities.
  • Lidar History: A short lidar history contains more useful information than only the current scan, while longer histories eventually provide no improvement or degrade performance.The evaluated windows were 0, 0.25, 0.5, 1, and 2 seconds.
  • Goal Position: The sub-goal is selected 2 m ahead along the nominal path using pure pursuit instead of feeding the final goal directly.This supports traversal of long paths through complex, non-convex environments and keeps the sub-goal distance constant for normalization.

D. Network Architecture

The architecture fuses lidar history and pedestrian kinematics before combining their extracted features with sub-goal information. Its handcrafted intermediate representations support flexible fusion and are evaluated against alternative fusion structures.

  • Network architecture: The feature extractor fuses historical lidar observations with pedestrian kinematics, then combines extracted features with sub-goal information.The actor and critic modules are built from fully connected layers.
  • Fusion ablation: Early fusion produced a network about half the size of the Late structure, nearly doubled processing rate, and slightly improved RMSE and EVA.The comparison used the same dataset and supervised training procedure.
  • Representation design: The architecture uses handcrafted features as intermediate representations rather than the learned features typical of learning-based algorithms.The paper identifies this as the key difference from other neural-network control policies.
  • Reward components: The reward combines goal progress, collision-related penalties, directional smoothness, and active steering toward collision-free sub-goals.The formulation addresses speed and safety as competing navigation objectives.
  • Velocity-obstacle reward: The velocity-obstacle reward term adds first-order pedestrian velocity information to guide proactive collision avoidance and goal-directed motion.Earlier approaches are described as relying mainly on passive, position-based collision avoidance rewards.

2) Passive Collision Avoidance:

The active-heading procedure extends velocity obstacles to select a desired direction that remains collision-free while moving toward the sub-goal. It uses tracked pedestrian motion and sampled candidate directions.

  • Velocity obstacles: The velocity obstacle represents robot velocities that would cause a collision with a moving obstacle at some future time.The construction begins by defining a special occupancy region around the pedestrian.
  • Direction search: Algorithm 1 samples candidate directions using tracked pedestrians, the robot’s linear velocity, and a specified number of samples.The search returns an optimal direction angle.
  • Collision cones: The collision cone contains relative velocities that will cause a future collision, allowing the robot to identify unsafe heading angles.These angles are computed for moving pedestrians tracked by MHT.
  • Direction selection: The selected desired direction moves toward the goal while remaining collision-free.The search combines collision-cone constraints with the sub-goal direction.

F. Deep Reinforcement Learning Algorithm

The policy is trained with proximal policy optimization, selected for implementation simplicity and competitive performance relative to other reinforcement-learning algorithms. PPO is also used widely in navigation studies.

  • Algorithm rationale: PPO is chosen because it is simpler to implement while achieving comparable or better results than A2C, TRPO, and ACER.The passage also notes prior navigation studies using PPO successfully.

IV. CROWDED DYNAMIC ENVIRONMENTS

The evaluation uses 3D Gazebo and PEDSIM simulations followed by Turtlebot2 hardware experiments. The setup covers training, testing, and reported results.

  • Experimental setup: The experiments combine Gazebo simulation with the PEDSIM pedestrian library and real-world tests on a Turtlebot2 robot.The section states that the experiments cover setup, training procedure, and results.

1) Robot Configuration:

The experiments use a Turtlebot2-based simulation and hardware setup spanning structured environments, varied crowd densities, and unseen test locations. Policies are trained in the Lobby world with 34 pedestrians and evaluated using navigation performance metrics against multiple baselines.

  • Robot configuration: The Turtlebot2 uses a ZED stereo camera, Hokuyo lidar, and a maximum velocity of 0.5 m/s.The camera has a 90° field of view and 0.3–20 m depth range; the lidar has a 270° field of view and 0.1–30 m range.
  • Simulation environments: The main Gazebo Lobby environment is approximately 25 × 10 m and contains static obstacles including chairs, tables, a security desk, waste bins, and pillars.Additional simulated environments include the approximately 20 × 10 m Autolab replica and other test worlds.
  • Training and evaluation: Training uses the Lobby environment with 34 pedestrians, while testing varies crowd density and evaluates unseen environments including Autolab, Cumberland, Square, and Freiburg.The evaluation includes Lobby trials with 5–55 pedestrians and unseen environments with 25–35 pedestrians.
  • Hardware evaluation: Hardware tests use a Turtlebot2 equipped with an NVIDIA Jetson Xavier embedded computer running the deployed policy.The hardware evaluation covers an indoor hallway, indoor lobby, and outdoor hallway, with experiments across crowd sizes and environments.
  • Evaluation protocol: Policies are compared using success rate, average time, average length, and average speed across trials with 25 goal points per environment.The comparison includes DRL, DRL-VO, A1-RD, A1-RC, a CNN policy, and DWA, with baseline policies used without retraining or parameter tuning.

1) Different Crowd Densities:

DRL-VO maintains strong navigation performance across crowd densities and several unseen environments, achieving a favorable balance between collision avoidance and speed. Its performance declines in larger open spaces and under failure conditions involving dense or abruptly changing pedestrian motion.

  • Different crowd densities: DRL-VO achieves the highest success rate and fastest average speed across tested crowd densities, with advantages increasing as pedestrian density rises.Its average speed remains nearly constant across crowd densities, while its success rate exceeds the standard DRL policy at each tested density.
  • Different unseen environments: In unseen environments, DRL-VO generally achieves higher success rates and slightly higher average speeds than DRL, except for slightly lower speed in Square world.The tested environments are Autolab, Cumberland, Freiburg, and Square, with 25 and 35 pedestrians.
  • Different unseen environments: DRL-VO has the highest success rate in the Lobby, Autolab, and Freiburg worlds and generally produces faster, more consistent speeds and shorter paths than competing policies.DWA is the exception regarding path length, while DRL-VO still demonstrates strong generalization to unseen environments.
  • Limitations: Success rates drop in Cumberland and Square, where larger open spaces may expose a mismatch between Lobby training and less structured environments.The paper identifies broader training environments or greater suitability of the VO-based reward for structured spaces as possible explanations.
  • Output velocity distribution: DRL-VO generates about 90% of its linear velocity commands near the upper end of the range, supporting its consistently higher average speed.A1-RD produces over 40% of commands near the lower end, reflecting its stop-and-wait behavior; DWA produces near-zero output in particularly crowded regions.
  • Output velocity distribution: DRL-VO produces fewer high-angular-velocity commands because its reward includes a path-smoothness term.Most policies issue small angular updates, while CNN and A1-RC are notable outliers in their angular velocity distributions.
  • Failure cases: Failure cases include difficulty finding collision-free directions in very dense crowds and handling sudden pedestrian movements.The reported causes include imperfect pedestrian simulation, missed tracks under occlusion, and limited safety guarantees for the learned black-box policy.

D. Hardware Results

Hardware experiments show that DRL-VO transfers from simulation to real robots and handles varying crowd densities across indoor and outdoor environments without retraining. The policy maintains safe navigation and real-time operation while pedestrians behave in ways not represented during training.

  • Real-world deployment: The Turtlebot2 used DRL-VO trained in Gazebo directly, without fine-tuning, with up to 60 FPS inference on a Jetson AGX Xavier.The policy has 32.137 M parameters and runs faster than the sensors’ 40 Hz rate.
  • Different crowd densities: 131.24 m at 0.41 m/s was completed safely in a low-density indoor hallway, while 137.28 m at 0.43 m/s was completed without collisions at medium density.The experiments used a sequence of 15 goal points under different crowd densities.
  • Different crowd densities: 133.40 m was traversed safely through a high-density hallway containing moving and stationary pedestrians.The policy actively avoided moving pedestrians and static people standing or sitting in the hallway while reaching predefined goals.
  • Different environments: 68.96 m at 0.37 m/s was completed through a structured indoor lobby, with no significant navigation difference from the more open hallway environment.The lobby contained chairs, tables, bicycles, trash bins, many pedestrians, and narrow passageways.
  • Different environments: 102.73 m at 0.43 m/s was completed without collisions in a dense outdoor hallway, supporting operation across indoor and outdoor settings.The outdoor hallway had no static obstacles but many pedestrians, and the authors describe the policy as robust to both environment types.
  • Unmodeled pedestrian behavior: Pedestrians sometimes stopped to observe the robot, although this curiosity behavior was absent from training data and the robot handled it successfully.The authors also report modeling stronger repulsion from the robot than between pedestrians.

V. HIGHLY CONSTRAINED ENVIRONMENTS

The policy was adapted to highly constrained static environments and a different robot platform with limited system changes. Deployment required odometry-only navigation, modified lidar and goal inputs, and a velocity limit near obstacles.

  • Challenge setting: The BARN challenge tested DRL-VO in unknown static environments and on a Jackal robot with different speed and lidar characteristics from the Turtlebot2.Jackal maximum velocity was 2 m/s versus 0.5 m/s for Turtlebot2; its lidar range was 10 m versus 30 m.
  • System adaptations: Unknown BARN maps required removing AMCL and operating with odometry only, while the stationary-obstacle setting set pedestrian inputs to zero.The adapted lidar history used the UTM-10LX’s full 270° FOV and an 80 × 80 representation.
  • System adaptations: A 1 m look-ahead sub-goal replaced the 2 m setting to help the robot avoid obstacles in highly constrained environments.The shorter look-ahead was used when generating the goal input for the DRL-VO network.
  • Velocity adaptation: Directly mapping normalized policy velocity to the Jackal caused aggressive behavior and frequent obstacle bumps.The problem was attributed to transferring a policy trained on the slower Turtlebot2 to the faster Jackal.
  • Velocity adaptation: The maximum velocity mechanism allows 2 m/s only when no obstacle is nearby and slows the robot near obstacles using df = 2.2 m.Here, obs is the closest-obstacle distance at time t, and the slower limit matches the training velocity near obstacles.

C. Results

BARN results show strong simulated generalization but weaker hardware performance in narrow, sharply turning environments. The policy transferred from a crowded dynamic Turtlebot2 setting to static environments and a Jackal platform, yet its training scope limited performance in the hardest physical layouts.

  • Simulation: 0.2415 was the average highest navigation score among 11 policies in the BARN simulation competition.DRL-VO scored 0.25 on every map it completed, while other competitors ranged from 0.1627 to 0.2334.
  • Hardware: The physical competition result was third place, with 2 of 9 runs completed successfully.Both successful runs occurred in the first environment, which had relatively wide passages.
  • Hardware: The other two physical environments were not completed because sharp turns and gap traps lacked reasonable nominal paths, while narrow passages exposed insufficiently precise and aggressive control.The policy had been trained without narrow passages and on the smaller Turtlebot2 robot.
  • Future improvements: The authors planned map-based navigation and training across multiple environments to address nominal-path and narrow-passage failures.These changes target the limitations observed in the physical competition.
  • Generalization: The BARN experiments showed transfer to a new static-navigation task, a different robot platform, and different sensor configurations after training in one crowded dynamic lobby with 34 pedestrians.The authors attribute this generalization in part to preprocessed data representations.

A. Data Representations

The policy uses handcrafted intermediate representations and a velocity-obstacle reward to support efficient learning and transfer from simulation to real-world navigation. These choices address the complexity of crowded dynamic environments while balancing collision avoidance, goal progress, and computational constraints.

  • Data Representations: Handcrafted intermediate features summarize environmental structure and pedestrian kinematics while reducing dependence on photorealistic simulation or real training data.The representation uses pooled lidar information and relative motion data to abstract away unnecessary scene complexity.
  • Data Representations: The abstracted input structure enables faster simulation iterations while supporting generalization from simulated environments to real-world settings without policy modification.The simulator can use flat textures and simplified rigid-body pedestrians because the policy operates on consistent intermediate representations.
  • Training Environment: Real-world pedestrian reactions remain a limitation because people may avoid, observe, follow, or flee from the robot, whereas the training model uses a simplified repulsive social force.The authors identify richer human-robot interaction modeling as a direction for future work.
  • Reward Design: The velocity-obstacle reward guides the robot toward the goal while proactively avoiding potential future pedestrian collisions.Velocity obstacles provide guidance from model-based collision-avoidance principles without directly generating the control action.
  • Data Representations: The policy combines a short lidar history, pedestrian position and velocity, and a sub-goal point in the robot’s local coordinate frame.This representation is designed for crowded dynamic environments and improves robustness to localization errors.
Loading 2301.06512v2…