PaperlayerView exact revision on arXiv

Source-linked AI summary

Social-WM: Safety-Aware Latent World Models for Robot Social Navigation

Zhihao Zheng, Mooi Choo Chuah

arXiv:2609.40177v1cs.RO

TL;DR

Social navigation requires robots to reach goals while avoiding collisions and respecting pedestrian personal space, but whether a nominal action is actually realizable under physical and social constraints remains an open question. Social-WM learns action-conditioned latent futures and realized motion, then uses their discrepancy to evaluate candidate actions before execution. It reports improved social-safety outcomes on Social-HM3D and strong results under zero-shot transfer to Social-MP3D.

  • Problem

    Social navigation must balance reaching goals with avoiding collisions and respecting pedestrian personal space, while whether nominal actions are realizable under physical and social constraints remains underexplored.

    Source: §I · p. 1

  • Method

    Social-WM learns action-conditioned latent futures from observed outcomes and realized motion, then uses inverse dynamics to assess candidate actions imagined by its planner.

    Source: §I · pp. 1–2

  • Results

    Social-WM achieves the highest SR (63.77%) and PSC (94.37%) and reduces H-Coll to 21.67% on Social-HM3D; on Social-MP3D it achieves the highest SR (54.15%) and lowest H-Coll (23.34%).

    Source: C. Quantitative Results · p. 6

  • Takeaways & Limitations

    Its goal-independent dynamics support position- and image-goal navigation, with reported safety gains using RGB and offline training without explicit tracking, privileged human state, or online reinforcement learning.

    Source: §I · p. 2

    Many remaining failures are timeouts: the model reasons mainly about short-horizon safety and action realizability, while long-horizon route selection remains unchanged.

    Source: J. Limitations and Failure Cases · p. 8

Abstract

from arXiv · show

Safe social navigation requires a robot to anticipate not only the future consequences of its actions, but also whether a nominal action can actually be executed under surrounding physical and social constraints. We present Social-WM, an efficient latent world-model planning framework trained from egocentric RGB video sequences. Our key observation is that social-navigation experience contains a systematic discrepancy between the nominal action and the realizable action: a nominal forward action may be fully executed in free space, but needs to be constrained when heading towards a pedestrian or obstacle. Social-WM learns these safety-relevant consequences directly through action-conditioned future prediction, where the target is the actual observed future following each command. We further introduce a realizable inverse-dynamics objective that associates observed latent transitions with the action actually realized rather than the nominal one. At deployment, candidate actions are imagined through the latent world model, and the inverse dynamics model estimates their realizability; nominal--realizable discrepancy then provides a safety signal before execution. The learned dynamics and realizability model remain goal-independent and support both position- and image-goal navigation. On Social-HM3D, Social-WM achieves 63.77% success while reducing human collisions to 21.67%, and maintains strong performance under zero-shot transfer to Social-MP3D, without explicit pedestrian tracking, privileged human state, or online reinforcement learning.

I. INTRODUCTION

Social navigation couples goal seeking with pedestrian safety and constraints on which actions can be executed. Social-WM predicts action-conditioned futures and estimates realizable motion, using nominal–realizable discrepancy to screen candidates before execution.

I. INTRODUCTION

  • Prior work predicts pedestrian trajectories or future scene and interaction representations, but whether nominal actions are realizable remains underexplored, [3].Source: §I · p. 1
  • Social-WM learns action-conditioned latent futures from observed outcomes and trains inverse dynamics to recover motion actually realized rather than the issued command.This grounds its learned dynamics in physically achievable motion and treats nominal–realizable discrepancy as a safety signal.Source: §I · pp. 1–2
  • At deployment, candidate action chunks are imagined and evaluated for goal progress and realizability, with large nominal–realizable discrepancies indicating likely blocked or inadmissible actions.The goal-independent dynamics and realizability models support both position-goal and image-goal navigation.Source: §I · pp. 1–2
  • Social-WM achieves competitive navigation success while reducing human collisions and improving personal-space compliance on Social-HM3D and zero-shot Social-MP3D.It uses RGB observations and offline training without explicit pedestrian tracking, privileged human state, or online reinforcement learning.Source: §I · p. 2

II. RELATED WORK

The related work spans social navigation, latent world models, and inverse dynamics. Social-WM distinguishes itself by scoring imagined action transitions according to whether commanded motion matches motion that is physically and socially realizable.

A. Social Robot Navigation

  • Social-WM uses imagined latent transitions directly for continuous action selection, unlike approaches that predict people or feed imagined features into a policy, [3].Earlier work includes social-force and reciprocal collision-avoidance methods, future occupancy or motion prediction, and interaction-aware approaches –.Source: A. Social Robot Navigation · p. 2

B. World Models for Embodied Decision-Making

  • Social-WM extends efficient latent prediction with explicit action realizability, scoring imagined action chunks and rejecting commands inconsistent with predicted realizable motion.This contrasts with NavThinker, which uses imagined features to support an online-trained policy [3].Source: B. World Models for Embodied… · p. 2

C. Inverse Dynamics and Multimodal Action Generation

  • Social-WM supervises inverse dynamics with odometry-derived realizable motion, then applies it to imagined transitions so command–motion discrepancy can guide candidate selection.Standard inverse dynamics instead associates observed transitions with the recorded action,.Source: C. Inverse Dynamics and Multimodal… · p. 2

III. METHODOLOGY

Social-WM treats the gap between commanded and safely executable motion as a signal in partially observed, human-populated indoor navigation. Its latent world model predicts observed outcomes and supports planning that evaluates both goal progress and action realizability.

A. Problem Formulation

  • When an obstacle or pedestrian constrains a command, the realizable motion can differ from the nominal action, making their discrepancy safety-relevant.The setting uses a forward-facing RGB camera and plans over H-step action chunks.Source: A. Problem Formulation · p. 2

B. Overview

  • The latent world model is trained from transitions whose prediction target is the RGB observation actually following a nominal action.This lets the dynamics learn when a command needs to be stopped or altered under safety constraints.Source: B. Overview · p. 3
  • Social-WM jointly grounds future predictions with inverse dynamics that recovers realizable rather than nominal actions, then evaluates imagined candidates for goal progress and realizability.The same goal-independent dynamics support closed-loop social-navigation planning.Source: B. Overview · p. 3

C. Safety-Aware Latent World Model

Social-WM predicts future observations in latent space using the actual outcome of each nominal command as its target. A realizable inverse-dynamics objective trains the model to associate latent transitions with the motion that occurred, supporting safety evaluation before execution.

C. Safety-Aware Latent World Model

  • A frozen DINOv2-small encoder converts each RGB observation into spatial patch tokens, which are pooled and projected into a latent representation.The class token is discarded before adaptive pooling.Source: C. Safety-Aware Latent World Model · p. 3
  • The future predictor is conditioned on nominal actions but targets the latent encoding of the actual observed future, including stopped or constrained outcomes.It predicts in latent space without reconstructing RGB or explicitly estimating depth, pedestrian trajectories, or human states.Source: C. Safety-Aware Latent World Model · p. 3
  • Training combines one-step teacher-forced prediction, representation regularization, and multi-step open-loop prediction to reduce accumulated rollout error.The multi-step objective is intended to reduce prediction error during the planner's multi-step rollouts.Source: C. Safety-Aware Latent World Model · p. 3

D. Realizable Inverse Dynamics

  • Realizable inverse dynamics associates observed latent transitions with the action actually executed, rather than the nominal command.The realizable action is recovered from consecutive robot poses; predicting it requires the latent transition to preserve safety-related scene information.Source: D. Realizable Inverse Dynamics · pp. 3–4
  • At deployment, inverse dynamics predicts the realizable action from imagined futures so the planner can identify potentially unsafe candidates before execution.Future prediction captures command consequences, while inverse dynamics explicitly associates each transition with realizable motion.Source: D. Realizable Inverse Dynamics · p. 4

E. Trajectory Proposal

A goal-independent proposal model supplies plausible action-chunk alternatives; the world model then imagines them for evaluation. The default proposal model is a CVAE that can generate diverse local behaviors.

E. Trajectory Proposal

  • A lightweight generative model conditioned on recent latent and action history proposes K plausible H-step action chunks.Its stochastic input introduces variation in the proposed chunks.Source: E. Trajectory Proposal · p. 4
  • The default CVAE generates diverse local behaviors such as proceeding, turning, waiting, or bypassing, while leaving goal-progress and realizability evaluation to later stages.The proposal model itself is goal-independent.Source: E. Trajectory Proposal · p. 4
  • The CVAE is trained with standard reconstruction and KL objectives.Source: E. Trajectory Proposal · p. 4

F. Safety Evaluation and Selection

Social-WM imagines each candidate through the world model and uses nominal–realizable action discrepancy as a feasibility signal. Among feasible candidates, selection is goal-conditioned and execution repeats in a closed loop.

F. Safety Evaluation and Selection

  • A large residual between nominal and inverse-dynamics-predicted realizable actions indicates a candidate is likely constrained, and action realizability is treated as a feasibility requirement.For example, a predicted stopped future for a forward command toward a pedestrian yields greater discrepancy.Source: F. Safety Evaluation and Selection · p. 4
  • The controller selects the lowest-cost candidate from the safe set.Only the first action of the selected chunk is executed before the robot observes a new RGB frame and repeats planning.Source: F. Safety Evaluation and Selection · p. 5
  • Candidates are ranked by goal-conditioned cost: position goals use closest target distance, while image goals use similarity to the goal latent.Goal representation is orthogonal to the safety evaluation, which is independent of the goal.Source: F. Safety Evaluation and Selection · p. 5
  • Candidate generation and latent rollouts are evaluated in parallel during the repeated propose–imagine–evaluate–select cycle.This retains the compact backbone's efficiency while providing multi-step foresight.Source: F. Safety Evaluation and Selection · p. 5

A. Experimental Setup

Social-WM is evaluated on Social-HM3D and zero-shot Social-MP3D, with comparisons to navigation baselines and measurements of success, efficiency, personal-space compliance, and human collisions. Its implementation learns from trajectories containing blocked or constrained motion, and qualitative results show it slowing or waiting near pedestrians.

A. Experimental Setup

  • The evaluation follows NavThinker’s single-robot protocol and reports SR, SPL, PSC, and H-Coll against rule-based, reinforcement-learning, and future-aware baselines.The predictor uses history T=8 and future H=8, and the planner selects from K=8 candidates; the visual encoder is frozen.Source: A. Experimental Setup · p. 5, B. Implementation Details · pp. 5–6

B. Implementation Details

  • Training uses 200 Social-HM3D scenes, 11,676 episodes, and 463,645 transitions; a keep-out gate rejects unsafe nominal actions, producing blocked or constrained motion in 18.1% of transitions.Source: B. Implementation Details · p. 5

D. Qualitative Results

  • In qualitative comparisons, Social-WM slows or waits to maintain pedestrian separation where unsafe interactions show insufficient clearance.Source: Figure 4 · p. 6

C. Quantitative Results

  • On Social-HM3D, Social-WM achieves the highest SR (63.77%) and PSC (94.37%) and reduces H-Coll to 21.67%; on zero-shot Social-MP3D, it achieves the highest SR (54.15%) and lowest H-Coll (23.34%).Social-MP3D SPL is 44.32% and PSC is 93.95%.Source: C. Quantitative Results · p. 6

COMPARE FINAL-GOAL AND INTERMEDIATE-SUBGOAL SETTINGS

Intermediate subgoals improve navigation success and efficiency while safety metrics remain similar across goal variants. The planner comparison favors CVAE overall, although diffusion leads on SPL and PSC; qualitative examples contrast Social-WM’s cautious responses with baselines’ continued motion into pedestrian-occupied regions.

COMPARE FINAL-GOAL AND INTERMEDIATE-SUBGOAL SETTINGS

  • Habitat-official and Falcon continue goal-directed actions into pedestrian-occupied regions, while Social-WM slows, waits, or adjusts when intended progress is not safely realizable.Source: COMPARE FINAL-GOAL AND… · p. 6

E. Effect of Navigation Goals

  • Image subgoals raise SR from 42.91% to 59.52% and SPL from 27.07% to 35.23%; position subgoals raise SR from 63.77% to 68.65% and SPL from 49.62% to 58.72%.PSC and H-Coll remain similar across final-goal and subgoal variants; position guidance outperforms image guidance, especially without intermediate subgoals.Source: E. Effect of Navigation Goals · p. 6

F. Effect of Trajectory Planner

  • CVAE achieves 59.52% SR and 27.04% H-Coll; diffusion leads SPL (40.15%) and PSC (94.08%), while MPC is slower and less effective.The comparison keeps Social-WM dynamics, safety evaluation, and goal settings fixed.Source: F. Effect of Trajectory Planner · pp. 6–7

G. Effect of Realizable-Action Learning

Realizable-action learning and its use in safety evaluation provide complementary benefits, and the nominal–realizable residual strongly distinguishes blocked from free execution. The latent-representation comparison favors DINOv2-Patch for pose decodability and goal ranking.

G. Effect of Realizable-Action Learning

  • Inverse-dynamics supervision raises SR from 51.33% to 53.08% and SPL from 26.49% to 30.16%, while H-Coll falls from 44.97% to 42.12%; discrepancy-based safety evaluation further cuts H-Coll to 31.33%.The full model reduces human collisions by 13.64% over the latent-WM baseline.Source: G. Effect of Realizable-Action Learning · p. 7
  • Nominal–realizable residuals distinguish blocked from free transitions with AUROC 0.961, while free-transition realizable-action prediction has MAE 0.10.The diagnostic evaluates 22,512 transitions from 584 held-out episodes, including Easy and Hard subsets.Source: G. Effect of Realizable-Action Learning · p. 7, Table V · p. 7

H. Effect of Latent Representation

  • DINOv2-Patch is the only representation with positive R2 on all pose dimensions and reduces Err@Min from 0.659 m to 0.320 m and Regret from 0.422 m to 0.083 m versus LeWM-CLS.The comparison evaluates pose decodability and latent-space goal-ranking quality on frame pairs.Source: H. Effect of Latent Representation · p. 7, Table VI · p. 7

I. Efficiency and Latency

Social-WM evaluates imagined actions for realizability and plans efficiently, while maintaining competitive navigation performance. Remaining failures center on long-horizon route selection and image-goal guidance.

I. Efficiency and Latency

  • 58M parameters versus LeWM’s 18M; most added size comes from the frozen 22M DINOv2 encoder, while the inverse-dynamics head and CVAE are 2.4M and 5M.Source: §I · p. 7
  • The default planner takes 51 ms per step, versus 126 ms for Diffusion Policy and 139 ms for LeWM-style MPC, making it approximately 2.5× and 3× faster, respectively.Source: §I · p. 8

J. Limitations and Failure Cases

  • Persistent blockage can still cause timeouts: Social-WM avoids unsafe motion but cannot determine whether to wait or seek an alternative route.Remaining failures are dominated by timeouts associated with long-horizon route selection.Source: J. Limitations and Failure Cases · p. 8
  • Long-horizon image-goal navigation is constrained because unseen goal images offer little guidance and route-sampled subgoals become unreliable after large path deviations.The paper points to route-level replanning, spatial memory, and adaptive subgoal selection as future directions.Source: J. Limitations and Failure Cases · p. 8

V. CONCLUSION

  • Social-WM substantially reduces human collisions and improves personal-space compliance while maintaining competitive navigation success on offline RGB trajectories.These results include zero-shot transfer to Social-MP3D.Source: §V · p. 8

Related papers

Loading 2609.40177v1…