PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Policies for Short-Horizon Planning in Robot Crowd Navigation

Wendong Li, Jochen Garcke

arXiv:2608.27158v1cs.LG

TL;DR

Robot crowd navigation needs policies that represent diverse, temporally consistent avoidance choices rather than isolated single-step actions. PDPO uses diffusion-generated action chunks with offline demonstration pretraining and online PPO fine-tuning, and reports improved success on bounded evaluation while identifying simulation and inference limitations.

  • Problem

    Dense, dynamic crowds offer multiple feasible avoidance maneuvers, but many reinforcement-learning policies use restrictive single-step action formulations.

  • Method

    PDPO pretrains a diffusion policy on collision-avoidance demonstrations, fine-tunes it online with PPO, and executes five-step action chunks in a receding-horizon manner.

  • Results

    84.7% success rate on Bounded CrowdNav, outperforming retrained CrowdNav++ by 11.7 percentage points and the single-action diffusion-policy ablation by 11.0 percentage points on average.

  • Takeaways & Limitations

    Diffusion-based action chunking supports locally coherent, diverse short-horizon navigation proposals, while bounded environments enable more faithful evaluation of crowd-aware behavior.

  • Takeaways & Limitations

    Evaluation uses simulated humans following fixed, non-reactive policies, and iterative denoising adds inference overhead for real-time use.

Abstract

from arXiv · show

Robot crowd navigation requires safe and efficient decision-making under dense, dynamic, and multimodal human--robot interactions. Existing reinforcement-learning methods typically output a single reactive action at each timestep, which limits their ability to represent diverse short-term avoidance strategies. We propose Planning Diffusion Policy Optimization (PDPO), an offline-to-online reinforcement-learning framework that uses a diffusion policy to generate short-horizon action chunks for crowd navigation. PDPO is first pretrained on collision-avoidance demonstrations and then fine-tuned online with PPO by treating the denoising process as an internal decision process. During execution, the policy generates a five-step action chunk and applies it in a receding-horizon manner. Furthermore, we observe an evaluation artifact in common crowd-navigation benchmarks: without explicit boundary constraints, learned agents may leave the valid domain and bypass dense crowds. To address this, we introduce a setting in which boundary violations are treated as collisions. Experiments show that PDPO obtains an improved success rate over strong baselines, and ablations demonstrate that action chunks are especially important for the modified bounded benchmark.

1 Introduction

Robot crowd navigation requires diverse, temporally consistent decisions in dense, dynamic, partially observable crowds. PDPO addresses restrictive single-step policies with diffusion-based action chunks and also corrects boundary-related benchmark artifacts.

  • Motivation: Dense crowds require diverse local maneuvers and temporal consistency across several future actions.Feasible choices include passing on different sides, yielding, or moving through temporary gaps.
  • Motivation: Many reinforcement-learning navigation policies output one action per timestep from a restrictive unimodal distribution.This formulation is poorly matched to multimodal avoidance behavior.
  • Contributions: PDPO is pretrained on collision-avoidance demonstrations and then fine-tuned online with reinforcement learning.This two-stage procedure initializes navigation with demonstrated behaviors before online adaptation.
  • Benchmark Design: Unconstrained benchmarks can let agents leave the valid region and bypass dense crowds, motivating bounded evaluation where boundary violations count as collisions.The modified setting is intended to provide a more faithful assessment of crowd-aware navigation.
  • Contributions: PDPO uses diffusion policies to represent non-Gaussian, multimodal avoidance behaviors that single-step Gaussian policies struggle to capture.The method brings diffusion-policy-based reinforcement learning to crowd navigation.
  • Contributions: PDPO generates short action sequences as local motion plans and executes them with receding-horizon replanning.Only the first action is executed before the next observation and plan.

2 Related Work

Prior crowd-navigation methods include reactive rule-based systems and learning-based policies, but commonly represent decisions at the single-action level. This work instead emphasizes diffusion-based modeling of multimodal short-horizon action sequences.

  • Rule-based Crowd Navigation: Rule-based methods such as ORCA and Social Force compute collision-avoidance actions from predefined local interaction rules.They can be conservative in dense crowds, causing freezing behavior.
  • Learning-based Methods: Deep value-based reinforcement learning estimates state or state–action values but is less direct for continuous action control.Policy optimization methods such as PPO are widely used for continuous actions.
  • Learning-based Methods: Policy-optimization methods typically optimize single actions, leaving temporally coherent short-horizon plans implicit.Temporal coherence is instead encouraged indirectly through returns, value estimation, or reward design.
  • Learning-based Methods: CrowdNav++ improves safety and social compliance with human trajectory prediction but retains single-step, often unimodal Gaussian policies.It does not explicitly model distributions over short-horizon action sequences.
  • Diffusion Policies: Diffusion policies model complex action distributions and can generate multi-step action sequences through iterative denoising.This provides the representation used by the proposed approach.

3 Preliminaries

The problem is formulated as an MDP in which the robot observes itself and visible humans, predicts short-horizon human positions, and receives rewards for velocity actions. Diffusion policies replace unimodal Gaussian action distributions with iterative denoising.

  • Problem Formulation: Crowd navigation is modeled as an MDP with robot observations, human observations, transitions, rewards, and episode termination conditions.Episodes end at goal arrival, collision with a human or boundary, or maximum length.
  • Problem Formulation: Human observations include current positions and predicted future positions over a short horizon.The number of observed humans can vary by timestep.
  • Problem Formulation: The robot action is a two-dimensional velocity command, while humans follow a fixed policy unknown to the robot.The reward is defined as R(s_t, a_t), and transitions follow the environment dynamics.
  • Problem Formulation: The experiments use a constant-velocity predictor for short-horizon human position prediction, with other learning-based components fixed across methods except boundary handling.This isolates the effect of policy representation.
  • Diffusion Policy Background: The Gaussian policy parameterization is convenient for PPO but restricts conditional actions to a unimodal family.This limitation matters when multiple actions or action sequences are feasible under the same observation.
  • Diffusion Policy Background: A diffusion policy starts from Gaussian noise and applies reverse denoising transitions to produce an action or action sequence.The induced policy can represent complex, non-Gaussian, multimodal distributions.

4 Planning Diffusion Policy Optimization

PDPO conditions a diffusion policy on encoded crowd observations, generates five-step action chunks, executes the first action, and replans. It combines demonstration pretraining with PPO fine-tuning over the denoising process.

  • Planning Diffusion Policy Optimization: PDPO encodes the current crowd observation into a latent representation, samples a five-step action chunk, executes its first action, and replans.This creates closed-loop short-horizon planning under changing crowd observations.
  • Crowd Observation Encoding: The crowd encoder uses human-feature embeddings, masked self-attention, and attention pooling to condition the diffusion policy.Visibility masks exclude unobservable humans from attention computation.
  • Diffusion Action Chunk Policy: Action chunks represent local motion explicitly and can model multimodal sequence-level avoidance patterns.Examples include passing on one side, yielding, or moving through a temporary gap.
  • Diffusion Action Chunk Policy: Only the first chunk action is applied before a new observation triggers another sampled chunk.Receding-horizon execution combines temporal coherence with responsiveness.
  • Offline Demonstration Pretraining: PDPO first behaviorally clones collision-avoidance demonstrations and then refines the policy through online reinforcement learning.The demonstrations provide feasible initial behaviors for safer learning.
  • Offline Demonstration Pretraining: Expert chunks are constructed from five consecutive velocity commands collected by rolling out ORCA in simulation.The chunk length is H = 5.
  • Online Reinforcement Learning Fine-tuning: The online stage fine-tunes the diffusion policy with PPO by treating reverse denoising as an internal MDP.Denoising transitions and log probabilities are stored, and the environment reward and advantage are assigned to the denoising process.

5 Experiment

Experiments compare PDPO with rule-based and learning-based baselines under original and bounded CrowdNav protocols. Results show that boundary handling changes evaluation outcomes, while short-horizon action chunks drive PDPO’s strongest gains.

  • 5.1 Simulation Environment: The simulator contains one robot and 20 humans in a 12×12 m2 workspace, with random positions and goals and dynamic human flows.Humans follow ORCA for human-human collision avoidance and independently change goals with probability 0.5 each second.
  • 5.1 Simulation Environment: Bounded CrowdNav adds workspace walls and treats boundary violations as collisions, preserving the original crowd-generation process, robot dynamics, and evaluation metrics.The modified setting removes shortcut behaviors caused by leaving the valid workspace.
  • 5.2 Baselines and Training: PDPO is compared with ORCA, CrowdNav++ using constant-velocity prediction, and a diffusion-policy ablation that generates one action per timestep.Learning-based methods use the same five-timestep constant-velocity human-motion predictor and training budget.
  • 5.3 Results on the Original Benchmark: 90.6% success: PDPO outperforms the reported CrowdNav++ result by 3.6 percentage points on the original benchmark.Action chunking raises success from 86.2% to 90.6%, while the single-action diffusion-policy variant performs comparably to CrowdNav++.
  • 5.3 Results on the Original Benchmark: 16.1% of successful CrowdNav++ trajectories leave the nominal workspace, compared with 18.7% without action chunks and 19.8% for PDPO.These out-of-bounds successes motivate Bounded CrowdNav as the controlled evaluation setting.
  • 5.4 Results on Bounded CrowdNav: 84.7% success: PDPO exceeds retrained CrowdNav++ by 11.7 percentage points and the single-action ablation by 11.0 percentage points on Bounded CrowdNav.PDPO also reduces intrusion rate from 8.53% for CrowdNav++ to 6.81%, while using a longer path but shorter navigation time.
  • 5.5 Visualization of Action Chunks: PDPO generates 5-step action chunks at 0.25 s intervals, executes only the first action, and replans as pedestrians enter its observation range.The visualized chunks turn toward available gaps and become longer after the robot leaves the densest interaction region.

6 Limitations and Future Work

PDPO has limitations in inference efficiency, supervision of generated action sequences, and evaluation realism. Future work targets faster diffusion sampling, trajectory-level objectives, and more realistic crowd scenarios.

  • Iterative denoising adds inference overhead compared with single-step policies, motivating more efficient diffusion sampling for real-time use.
  • PDPO applies only the first action from each generated chunk, leaving trajectory-level objectives or consistency regularization as possible extensions.
  • Evaluation uses simulated humans following fixed policies that do not react to the robot, so it does not fully capture real human–robot interactions.

7 Conclusion

PDPO combines diffusion-based short-horizon action chunks with receding-horizon execution to represent diverse local choices while remaining responsive in crowds. Results and benchmark analysis support action chunking and bounded evaluation as important elements of crowd-navigation assessment.

  • PDPO generates short-horizon action chunks with a diffusion policy and executes them in a receding-horizon manner for responsive crowd navigation.
  • Quantitative ablations and visualizations show that short-horizon action sequences produce locally coherent and diverse proposals under the same observation.
  • Missing boundary constraints can affect policy evaluation, motivating bounded environments for more faithful assessment of crowd-aware navigation.
  • PPO fine-tunes diffusion policies by treating denoising as an internal decision process alongside the environment MDP.

A.1 Diffusion Policy as an Augmented MDP

PDPO represents diffusion denoising as an augmented internal decision process while preserving the environment-level crowd-navigation MDP. PPO updates denoising transitions using credit assigned from the executed action and its environment reward.

  • The reverse denoising process forms an augmented MDP whose transitions can be optimized with policy gradients without changing the environment MDP.
  • PDPO generates an action chunk through iterative reverse diffusion, then executes only its first action before the environment transitions.
  • The multi-step denoising procedure is interpreted as internal policy decisions whose final output determines the environment action.
  • PPO is applied to individual denoising transitions after behavioral-cloning pretraining of the diffusion policy.
  • The environment reward arrives after action execution, so its advantage is assigned across the denoising transitions that produced that action.
  • Later denoising steps receive greater credit, with the final transition receiving the full environment-level advantage.
  • Behavioral-cloning pretraining predicts injected Gaussian noise on demonstration action chunks to initialize collision-avoidance behavior.

B Network Architecture and Hyperparameters

PDPO encodes observable human interactions and robot state into a latent representation that conditions both diffusion denoising and value estimation. Its experiments use chunked diffusion outputs, PPO-related hyperparameters, and reported trainable-parameter counts.

  • Crowd Observation Encoding: The observation encoder uses human-feature embedding, masked self-attention, and attention pooling to represent observable crowd interactions.
  • Crowd Observation Encoding: Robot-conditioned human–robot features are attention-pooled, then combined with robot state into latent vector z_t.
  • Crowd Observation Encoding: The latent representation conditions both the diffusion denoising network and the value function during online fine-tuning.
  • Diffusion Policy: The diffusion network denoises flattened action chunks conditioned on the observation latent and timestep embedding, with output dimension 2H.
  • Value Function: The value function shares the observation encoder and supplies the environment-level advantage used for denoising-step credit assignment.
  • Hyperparameters: Table 3 organizes environment, architecture, diffusion-model, and training hyperparameters used in the experiments.
  • Model Size: PDPO updates 1.049M trainable parameters, including a 0.848M actor and 0.201M critic, versus 2.503M for CrowdNav++ in the reported implementation.

B.5 Training Curves

On Bounded CrowdNav, PDPO’s online fine-tuning achieves higher and more stable training success than its single-action diffusion-policy ablation.

  • PDPO achieves higher and more stable training success rates than the single-action diffusion-policy ablation during online fine-tuning.The comparison is shown over the common training horizon on Bounded CrowdNav.
  • Figure 2 reports training success rate on the left and average training reward on the right.

C Additional Visualization of Action Chunks

PDPO generates coherent five-step action chunks that adapt to nearby pedestrians and represent multiple feasible short-horizon plans under the same observation.

  • PDPO generates a 5-step action chunk every decision step, with a 0.25 s control interval and a 1.25 s short horizon.
  • Generated chunks adjust toward available gaps, turn smoothly around approaching pedestrians, and shorten in dense regions for more cautious motion.After leaving the constrained area, the planned motion becomes longer again.
  • Under the same observation, PDPO samples multiple feasible chunks, including alternatives that pass through different gaps or slow near pedestrians.
  • This variability represents multiple plausible local plans while replanning as the crowd evolves.
Loading 2608.27158v1…