PaperlayerView exact revision on arXiv

Source-linked AI summary

Crowd-Robot Interaction: Crowd-aware Robot Navigation with Attention-based Deep Reinforcement Learning

Changan Chen, Yuejiang Liu, Sven Kreiss, Alexandre Alahi

arXiv:1809.08835v2cs.ROcs.LG

TL;DR

Crowd navigation methods often neglect Human-Human interactions and therefore inadequately represent dense-crowd effects on robot decisions. This paper introduces Crowd-Robot Interaction modeling with self-attentive pooling and joint Human-Robot/Human-Human interaction encoding. Experiments report time-efficient, state-of-the-art crowd navigation and qualitative reasoning about human importance.

  • Problem

    Existing reinforcement-learning navigation models simplify collective crowd effects and commonly model only one-way Human-Robot interaction, omitting Human-Human interactions.

  • Method

    The model extracts pairwise Human-Robot features, encodes Human-Human interactions through local maps, and aggregates neighbors with self-attention.

  • Results

    The approach outperforms state-of-the-art navigation methods in time-efficiency and task accomplishments while demonstrating qualitative reasoning about the importance of humans in a crowd.

  • Takeaways & Limitations

    Joint crowd interaction modeling supports anticipation of crowd dynamics and socially attentive navigation in dense scenes.

Abstract

from arXiv · show

Mobility in an effective and socially-compliant manner is an essential yet challenging task for robots operating in crowded spaces. Recent works have shown the power of deep reinforcement learning techniques to learn socially cooperative policies. However, their cooperation ability deteriorates as the crowd grows since they typically relax the problem as a one-way Human-Robot interaction problem. In this work, we want to go beyond first-order Human-Robot interaction and more explicitly model Crowd-Robot Interaction (CRI). We propose to (i) rethink pairwise interactions with a self-attention mechanism, and (ii) jointly model Human-Robot as well as Human-Human interactions in the deep reinforcement learning framework. Our model captures the Human-Human interactions occurring in dense crowds that indirectly affects the robot's anticipation capability. Our proposed attentive pooling mechanism learns the collective importance of neighboring humans with respect to their future states. Various experiments demonstrate that our model can anticipate human dynamics and navigate in crowds with time efficiency, outperforming state-of-the-art methods.

I. INTRODUCTION

Crowd navigation requires anticipating interacting humans rather than treating them as static obstacles or isolated robot-facing agents. The paper addresses these limits with a Crowd-Robot Interaction model that combines self-attention and Human-Human interaction modeling.

  • Traditional navigation methods treat moving agents as static obstacles or use one-step lookahead, producing short-sighted, unsafe, and unnatural behavior.
  • Integrating crowd trajectory prediction into decision-making remains challenging because dense crowds create complex, interacting, and probabilistic futures.
  • Existing reinforcement-learning methods simplify pairwise interactions and usually model only one-way Human-Robot effects, ignoring interactions within the crowd.
  • The proposed CRI model uses self-attention to aggregate pairwise features while jointly encoding Human-Robot and Human-Human interactions.
  • Experiments show that the model anticipates crowd dynamics, navigates time-efficient paths, outperforms state-of-the-art methods, and transfers effectively to a real robotic platform.

B. Problem Formulation

The task is formulated as sequential reinforcement-learning control for a robot moving toward a goal through n humans. Agent states, the robot’s goal-related state, transitions, and socially shaped rewards define the navigation problem.

  • The robot navigates toward a goal through a crowd of n humans in a sequential decision-making framework.
  • Each agent exposes position, velocity, and radius, while the robot additionally observes its goal position and preferred speed.
  • The robot’s commanded action is its instantly achieved velocity, represented as v_t = a_t.
  • The optimal value function maximizes expected discounted return, with preferred velocity used to normalize the discount factor for numerical reasons.
  • The reward function awards task accomplishment and penalizes collisions or uncomfortable separation distances.

C. Value Network Training

The value network is initialized from demonstrations and refined through reinforcement learning using temporal-difference updates, replay, and a fixed target network. Training queries true next states to reduce dependence on an approximate motion model.

  • The value network uses temporal-difference training with standard experience replay and a fixed target network.
  • Deep V-learning initializes the value and target networks from demonstrator experiences before iterative episode-based updates.
  • Each update forms a target from the immediate reward, discounted target-network value, and next joint state, then applies gradient descent.
  • The training procedure obtains the next state by querying the environment’s true value instead of using a linear motion approximation.
  • Accurate value estimation requires modeling crowd interactions because incomplete interaction modeling degrades performance in densely populated scenes.

III. APPROACH

The approach decomposes socially attentive navigation into interaction, pooling, and planning modules. It explicitly models Human-Robot and Human-Human interactions, aggregates them with self-attention, and estimates joint-state value for planning.

  • III. APPROACH: Human behavior in dense crowds motivates estimating each neighboring human’s relative importance based on likely future interactions.
  • Interaction: The Interaction module models Human-Robot interactions explicitly and encodes Human-Human interactions through coarse-grained local maps.
  • Pooling: The Pooling module combines interaction features into a fixed-length embedding using self-attention.
  • Planning: The Planning module estimates the value of the robot-and-crowd joint state for social navigation.
  • III. APPROACH: The complete pipeline extracts robot-human interactions, aggregates them, and evaluates the resulting joint state for navigation in crowds.

A. Parameterization

The robot-centric parameterization places the robot at the origin with the x-axis directed toward its goal, transforming robot and human states into this frame.

  • The robot is located at the origin, and the x-axis points toward its goal in the robot-centric frame.
  • The parameterization includes the robot’s distance to the goal and its distance to each neighboring human.These distances are defined as dg = ||p−pg||2 and di = ||p−pi||2.

B. Interaction Module

The interaction module explicitly models robot–human pairs while representing human–human effects with local maps, producing fixed-length pairwise interaction embeddings.

  • The module avoids explicit all-pairs human interaction modeling by combining robot–human pairwise features with coarse-grained local maps.Explicitly modeling every human pair has O(N^2) complexity, which is undesirable for scaling in dense scenes.
  • For each human, an L × L × 3 local-map tensor encodes neighboring humans’ presence and velocities around that human.The map is centered on human i, and each neighbor contributes a local state vector containing velocity components and an occupancy indicator.
  • An MLP embeds the robot state, human i’s state, and local map into a fixed-length vector e_i.The embedding function uses ReLU activations.
  • A subsequent fully connected ReLU layer transforms e_i into the pairwise interaction feature h_i between the robot and human i.

C. Pooling Module

The pooling module assigns attention scores using each person’s embedding and the mean crowd embedding, then forms a fixed-size crowd representation from weighted pairwise interactions.

  • Distance ordering alone is insufficient because a neighbor’s speed and direction also affect its potential influence.The pooling design therefore handles varying numbers of humans without assuming the closest neighbors matter most.
  • An MLP transforms each interaction embedding e_i, together with the mean embedding e_m, into an attention score α_i.The mean embedding is obtained by pooling the individual embeddings.
  • The final crowd representation c is a weighted linear combination of the pairwise interaction vectors h_i.The weights are produced from the attention scores through softmax normalization.
  • The resulting compact crowd representation is passed to a planning module that estimates the state value v.The value function is implemented as an MLP with ReLU activations.

E. Implementation Details

The implementation uses simulated circle-crossing environments, specified training procedures, discrete holonomic actions, and comparisons against three established navigation baselines in invisible and visible settings.

  • Training uses ORCA demonstrations for imitation learning and Adam-based reinforcement learning with specified learning rates, discounting, and ε-greedy exploration.Imitation learning uses 3k demonstration episodes for 50 epochs; reinforcement-learning exploration decays during the first 5k episodes.
  • The robot assumes holonomic kinematics and selects among 80 discrete actions formed by five speeds and 16 headings.
  • The simulation uses ORCA-controlled humans with Gaussian-sampled parameters in circle-crossing scenarios with randomized starting and goal positions.
  • The full LM-SARL model is compared with ORCA, CADRL, and LSTM-RL, while SARL removes the local map for ablation.The planning module is kept identical across the proposed model and RL baselines for fair comparison.
  • Evaluation includes invisible and visible settings, distinguishing cases where humans react only to one another from cases with mutual robot–human impacts.The invisible setting also removes the uncomfortable-distance reward penalty to isolate interaction reasoning.

B. Quantitative Evaluation

Across invisible and visible settings, the evaluation shows that attentive reinforcement-learning policies handle crowd interactions more effectively than the baselines. SARL and LM-SARL achieve strong navigation outcomes, while LM-SARL further improves navigation time, cumulative reward, or final reward depending on the setting.

  • Invisible Robot: In the invisible setting, ORCA fails because its reciprocal assumption is violated, while CADRL has the lowest success rate among reinforcement-learning methods.CADRL’s maximin aggregation considers only one interaction pair at a time, highlighting the need to account for all humans simultaneously.
  • Invisible Robot: SARL completes all invisible-setting test cases and substantially reduces average navigation time compared with direct aggregation and baseline methods.LSTM-RL achieves a higher success rate but still experiences occasional collisions and timeouts.
  • Invisible Robot: LM-SARL achieves the best invisible-setting results, outperforming SARL in navigation time and cumulative reward.The improvement is attributed to encoding interactions among humans, although the margin is not large.
  • Visible Robot: In the visible setting, ORCA achieves a high success rate and never enters humans’ comfort zones, but its short-sighted and conservative behavior yields low rewards.Table II includes discomfort frequency alongside success, collision, time, and reward, with ORCA’s collision and discomfort values set to zero by design.
  • Visible Robot: Visible-setting reinforcement-learning results resemble the invisible-setting results: SARL significantly outperforms baselines, and LM-SARL further improves the final reward.Human-Human interactions are not significant in every episode, diluting their quantitative effect while producing qualitative improvements.

C. Qualitative Evaluation

Qualitative evaluations show that SARL and LM-SARL use crowd-aware reasoning to select safer, shorter paths and prioritize humans likely to affect future navigation. LM-SARL further incorporates Human-Human interactions when estimating action values in dense scenes.

  • Trajectory comparison: SARL and LM-SARL identify a central shortcut that preserves distance from humans while reaching the goal quickly.CADRL passes aggressively, whereas LSTM-RL slows dramatically and incurs a long navigation time.
  • Attention scores: LM-SARL assigns low attention to humans walking away and the highest attention to human #3, who is most likely to approach the robot.Human #2 is closest, but #3 receives greater attention because of the predicted near-future interaction.
  • Attention scores: The attentive pooling module reasons about the relative importance of humans in a dense scene.
  • Value estimation: Baseline value estimates favor fast motion toward the goal despite danger from humans #1 and #3, while SARL slows down and waits safely.CADRL assigns its highest value to 120°, and LSTM-RL still overestimates high speeds near that direction.
  • Value estimation: LM-SARL selects a 200° turn to pass behind humans #1 and #3, indicating potential to reason about complex multi-agent interactions.The model encodes Human-Human interactions through local maps when estimating this action.

D. Real-world Experiments

The paper also examines its trained policy on a Segway robotic platform and concludes that the approach outperforms state-of-the-art navigation methods in time-efficiency and task accomplishment. The cited passage points to a video demonstration of the real-world experiment.

  • Real-world evaluation: The trained policy is evaluated in a real-world experiment on a Segway robotic platform.A video demonstration is provided at the cited YouTube link.
  • Conclusion: The paper reports that its approach outperforms state-of-the-art navigation methods in time-efficiency and task accomplishments.
Loading 1809.08835v2…