PaperlayerView exact revision on arXiv

Source-linked AI summary

Advantage-Driven Explicit Memory for Social Navigation

Yeonsoo Park, Mattia Racca, Guillaume Bono, Steeven Janny, Gianluca Monaci, Tomi Silander, Christian Wolf

arXiv:2608.25610v1cs.RO

TL;DR

The paper addresses the difficulty of preserving rare, high-cost social-navigation failures in policies that store behavior primarily in network parameters. It adds advantage-driven episodic memory to a recurrent PPO agent, retrieving hidden-state-indexed events and collecting test-time experiences; the reported results show improved in-distribution performance and online recovery in unseen social contexts.

  • Problem

    Parametric IL and RL risk diluting rare, high-error interaction signals, limiting preservation of critical social-navigation failures.

  • Method

    A recurrent PPO navigation agent stores hidden-state-indexed episodic events containing actions and saturated future values, selected using advantage or deployment-time surprise.

  • Results

    79.60% SR, a 3.44% absolute improvement over the 76.16% baseline, is achieved with external memory inactive; online memory reaches 60.00% SR and 52.16% SPL.

  • Takeaways & Limitations

    Explicit memory preserves rare critical interaction patterns, improves sample efficiency, and provides incremental behavioral adaptation without backpropagation.

Abstract

from arXiv · show

Robot policies are predominantly learned with classical parametric variants of imitation learning or RL, where training stores the agent's behavior exclusively in the policy's network parameters, putting a heavy burden on the representation learning algorithm. We propose a new navigation agent equipped with non-parametric memory which explicitly indexes prior steps leading to critical events. The advantages are twofold: first, it allows the policy to outsource some of its behavior into an explicit memory; second, it encourages a form of continual learning by allowing an agent to collect data from its testing episodes during deployment and therefore to better generalize to OOD situations. In the context of social navigation, we show that this improves the agent's capability to retain sparse, high-cost failures, such as human collisions. If the policy is trained in simulation, this also naturally addresses the sim-to-real gap, partially, by basing some of the decision making on real data. We integrate the explicit memory into a recurrent PPO architecture and use hidden states for memory retrieval to capture continuous spatiotemporal dynamics. The goal of exploiting rare, high-impact events is achieved by leveraging the RL agent's advantage signals. We train our agent in simulation with a combination of photorealistic rendering and non-visual crowd simulation and show that the agent is robust with respect to OOD social behavior.

I. INTRODUCTION

The paper introduces advantage-driven explicit episodic memory for social navigation, preserving rare interaction failures and supporting adaptation to distribution shifts through test-time experiences.

  • Parametric IL and RL can dilute high-error signals from rare, difficult navigation moments within policy weights.The motivation centers on edge cases such as blind corners and narrow doorways.
  • The method stores agent states linked to exact cumulative future rewards, creating an explicit situational cause-and-effect memory.This links actions to delayed consequences rather than storing only instantaneous observations.
  • Recurrent hidden states serve as memory keys, enabling retrieval based on continuous interaction dynamics rather than spatial snapshots.Retrieved interactions are aligned with the agent’s current situation.
  • Large-magnitude advantage signals select rare, high-impact experiences for storage and counter their catastrophic forgetting.The selection preserves both positive and negative critical events.
  • The explicit memory is organized as a Scorecard mapping candidate hidden states and actions to resulting utilities.Integrated action-outcome evaluation is intended to improve sample efficiency during training.
  • Test-time memory collection enables behavioral adaptation to OOD situations without additional gradient updates.Replacing training memory with testing experiences is described as mitigating distribution shifts.

II. RELATED WORKS

The agent extends recurrent end-to-end RL for human-aware navigation with human-trace encoding, explicit episodic memory, and PPO-based training in simulation.

  • Social navigation requires accounting for human expectations, intentions, and social norms alongside safety and efficiency.Rare high-cost events include collisions at blind corners or bottlenecks.
  • The base agent receives RGB, LiDAR-like, localization, goal, and surrounding-human observations before recurrent processing.A transformer-based human encoder forms global crowd features for the GRU.
  • The architecture uses a GRU policy, PPO advantage computation, attention-based retrieval, and GAE-based memory selection.Memory collection differs between offline advantage-based selection and online surprise-based selection.
  • The recurrent policy is fine-tuned end-to-end in simulation with PPO and a dense navigation reward.The reward includes success, path-distance, collision, and slack terms.
  • The external episodic memory is integrated into a PPO-GRU pipeline to store and retrieve critical spatiotemporal events.The architecture combines a recurrent base policy, memory retrieval, and causal memory collection.

A. Memory Augmentation

The memory augmentation module adds retrieved historical features to the recurrent policy, using hidden interaction states to query a fixed-capacity episodic buffer.

  • The method augments the recurrent agent with an external, non-parametric episodic memory module.Its three components are the recurrent base policy, spatiotemporal retrieval, and causal memory collection.
  • The previous GRU hidden state encodes temporally integrated interaction history and serves as the current context for querying memory.Retrieved memory features are concatenated with observation embeddings before GRU processing.
  • The memory buffer stores entries accumulated over past navigation episodes and uses a fixed-capacity FIFO queue of N = 512.The cited implementation notes that alternative discard mechanisms remain future work.

B. Spatiotemporal Memory Retrieval and Fusion

Spatiotemporal retrieval represents each memory entry with an internal state, action, and saturated future value, integrating historical interaction data into current decisions.

  • Each memory entry combines an internal state, executed action, and saturated future Q-value ¯Q_n ∈ [−1, 1] from a selected past step.This representation supports retrieval and integration of historical interaction data.

1) Memory Structure and Querying:

The method retrieves past experiences using recurrent hidden states as continuous spatiotemporal queries, then integrates the most relevant memories into the policy. An adapted projection space and orthogonal regularization support discriminative retrieval.

  • Memory querying: Recurrent hidden states serve as queries and keys, retrieving experiences according to continuous interaction dynamics rather than instantaneous observations.The current and stored states are projected into a shared retrieval space before similarity computation.
  • Memory querying: A two-layer MLP adapts the pretrained GRU representation to the retrieval objective while preserving the existing latent state representation.
  • Memory querying: Orthogonal regularization augments the PPO loss to improve retrieval-space discrimination and prevent representation collapse.The regularization is applied to projected internal states collected during rollout trajectories.
  • Memory querying: The retrieval mechanism selects the top-5 experiences by cosine similarity and normalizes their weights with softmax temperature scaling τ = 1.0.Non-selected experiences are masked out before normalization.

C. Construction of Causal Episodic Entries

Memory entries connect earlier hidden states and actions to later outcomes, prioritizing surprising events and shifting stored context backward to support preventive behavior. Deployment adds training-free online labeling for OOD environments, while critic estimates can become unreliable there.

  • Construction of Causal Episodic Entries: The method uses a k-step look-ahead mechanism to select and construct historical interaction entries for the memory buffer.
  • Construction of Causal Episodic Entries: Absolute advantage identifies surprising outcomes, and the top 5% of collected steps are added to memory after each training batch.The indicator captures outcomes that are either better or worse than expected under the current policy.
  • Construction of Causal Episodic Entries: With k = 6, the method stores context and action from six steps before a detected event to retrieve causal precursors early enough for preemptive maneuvers.
  • Construction of Causal Episodic Entries: Saturated Q-values use +1 for behaviors to emulate and −1 for actions to avoid, making interaction outcomes intuitive to the attention mechanism.
  • Construction of Causal Episodic Entries: In OOD environments, unseen states can cause critic hallucination and unreliable value estimates, producing erroneous memory labels.
  • Construction of Causal Episodic Entries: Deployment replaces privileged-information-based surprise labeling with critic temporal-difference changes or sparse terminal rewards.The sparse reward assigns +1 to successful collision-free completion and −1 to other terminal outcomes.
  • Construction of Causal Episodic Entries: Human supervisors can manually label interaction points, overriding or reinforcing behaviors through intuitive +1 and −1 memory scores.

3) Memory Maintenance and Sustainability:

Joint policy and encoder training can cause representation drift that degrades long-horizon retrieval, although FIFO replacement and similarity filtering suppress outdated entries.

  • 3) Memory Maintenance and Sustainability:: Representation drift may affect memory retrieval over long training horizons because the encoder is jointly trained with the policy.
  • 3) Memory Maintenance and Sustainability:: FIFO memory replacement and similarity-based retrieval filtering mitigate drift by naturally suppressing outdated memory entries.

A. Simulation Environment and Base Agent Setup

Experiments use Habitat-Sim with HM3D for PointGoal navigation in simulated dynamic crowds. Training uses progressively denser random-waypoint crowds, while OOD tests introduce fixed high density and novel or more variable movement patterns.

  • A. Simulation Environment and Base Agent Setup: The experiments use Habitat-Sim and the HM3D dataset for high-fidelity indoor PointGoal navigation with simulated dynamic crowds.Crowd motion is generated with the Recast Navigation library, and people do not react directly to the robot.
  • A. Simulation Environment and Base Agent Setup: Training crowd agents move between random waypoints with velocity and acceleration sampled from distributions informed by collected trajectory data.
  • A. Simulation Environment and Base Agent Setup: Curriculum learning increases maximum crowd density from 0.05 to 0.2 people per square meter, with episode density uniformly sampled below the current maximum.At ρmax = 0.2, the average crowd density is 0.1 people per square meter.
  • A. Simulation Environment and Base Agent Setup: The evaluation compares in-distribution and out-of-distribution scenarios defined by crowd complexity and density.
  • A. Simulation Environment and Base Agent Setup: ID testing matches final training with random-waypoint motion and density sampled from [0, 0.2], giving an expected density of 0.1.
  • A. Simulation Environment and Base Agent Setup: OOD testing holds density at ρ = 0.2 and introduces grouping, idling, and more divergent, erratic velocity and acceleration profiles.Performance uses SR, SPL, and SCT; safety uses environment collisions and human collisions.

B. The impact of humans on baseline performance

Human presence reduces the baseline agent’s navigation success, while advantage-driven training and explicit retrieval improve in-distribution performance. The trained memory buffer adds further gains beyond the base policy.

  • The baseline agent’s success rate drops when evaluated with humans, despite being trained with crowds.
  • The in-distribution evaluation compares baseline PPO-GRU, context-disabled Ours, and full-memory Ours configurations.
  • Ours w/ context=0 achieves an SR of 79.60%, improving over the baseline’s 76.16% by 3.44% absolutely.
  • Advantage-driven selection prioritizes high-impact events such as near-collisions, increasing sample efficiency during training.
  • Activating the trained memory buffer raises SR to 81.28%, exceeding the context-disabled configuration.

LEVERAGING TEST TIME SAMPLES

Under novel social behaviors, distribution shift substantially degrades the offline-trained agent, whereas an online memory buffer recovers performance using deployment-time experiences. This adaptation requires no backpropagation, and query latency remains low as memory grows.

  • Under OOD social behavior, baseline PPO-GRU reaches only 54.00% SR, while the offline-trained buffer achieves 56.40% SR.
  • The online buffer raises SR to 60.00% and SPL to 52.16%, a 3.60% SR improvement over the trained buffer.
  • Online adaptation reduces the human collision metric to 3.24 by incorporating recent negative labels from physical outcomes.
  • The online buffer provides immediate behavioral benefits without requiring backpropagation.
  • Average query time stays between 0.10 ms and 0.125 ms as memory capacity increases from 512 to 4096 entries.

1) Memory Query Features:

Memory query design strongly shapes navigation behavior: hidden-state representations with orthogonal loss achieve the best reported success, while raw or human-only queries trade efficiency for conservatism or collision avoidance. Online value scoring likewise favors physical-outcome labels over critic predictions in OOD deployment.

  • Memory Query Features:: The hidden state augmented with orthogonal loss achieves the highest navigation performance, with SR of 81.28%.The authors attribute this to disentangled metric representations, reduced collapse, and increased memory coverage.
  • Memory Query Features:: All Obs minimizes static collisions but reduces SR to 73.84%, as sensory noise drives an overly conservative policy.
  • Memory Query Features:: Human Feat yields the fewest human collisions at 2.24 but lowers SR to 73.72%, producing an avoidance-specialist strategy.
  • Online Labeling Strategies:: Critic-predicted values are unreliable in OOD environments, increasing human collisions to 4.00 during online memory collection.
  • Online Labeling Strategies:: Physical-outcome labels assign −1 to critical failures and +1 to successful navigation, balancing social safety with navigation efficiency.The rule-based labels provide a crude proxy for human preference during online collection.
Loading 2608.25610v1…