PaperlayerView exact revision on arXiv

Source-linked AI summary

VLM-Social-Nav: Socially Aware Robot Navigation through Scoring using Vision-Language Models

Daeun Song, Jing Liang, Amirreza Payandeh, Amir Hossain Raj, Xuesu Xiao, Dinesh Manocha

arXiv:2404.00210v3cs.RO

TL;DR

Social navigation requires contextually appropriate behavior, but human and environmental contexts are difficult to categorize, and language models face real-time latency challenges. VLM-Social-Nav combines perception with VLM-based scoring to guide a motion planner. In four scenarios, it improved average success and collision rates, and its user study score indicated the most socially compliant behavior.

  • Problem

    Contextually appropriate navigation is difficult because human behavior and environmental or task contexts are not easily categorized, while language models' latency challenges real-time social navigation.

    Source: §I · p. 1

  • Method

    VLM-Social-Nav detects key social entities and uses VLM-based scoring to produce a social cost that guides a bottom-level motion planner.

    Source: §I · p. 2

  • Results

    27.38% improvement in average success rate and 19.05% reduction in average collision rate were reported across four social navigation scenarios.

    Source: §I · p. 2, C. Quantitative Result · p. 7

  • Takeaways & Limitations

    The system was evaluated with a Turtlebot in four real-world social navigation scenarios and reduces dependence on extensive training datasets.

    Source: §V · pp. 7–8

    Experiments were conducted only indoors, and the authors identify outdoor navigation and scenarios involving multiple groups as future work.

    Source: E. Discussion · p. 7

Abstract

from arXiv · show

We propose VLM-Social-Nav, a novel Vision-Language Model (VLM) based navigation approach to compute a robot's motion in human-centered environments. Our goal is to make real-time decisions on robot actions that are socially compliant with human expectations. We utilize a perception model to detect important social entities and prompt a VLM to generate guidance for socially compliant robot behavior. VLM-Social-Nav uses a VLM-based scoring module that computes a cost term that ensures socially appropriate and effective robot actions generated by the underlying planner. Our overall approach reduces reliance on large training datasets and enhances adaptability in decision-making. In practice, it results in improved socially compliant navigation in human-shared environments. We demonstrate and evaluate our system in four different real-world social navigation scenarios with a Turtlebot robot. We observe at least 27.38% improvement in the average success rate and 19.05% improvement in the average collision rate in the four social navigation scenarios. Our user study score shows that VLM-Social-Nav generates the most socially compliant navigation behavior.

I. INTRODUCTION

Social navigation requires robots to interpret context and human behavior, yet learning-based methods often need substantial training data and VLMs face latency challenges. VLM-Social-Nav uses perception and VLM scoring to guide socially compliant actions, with improved results across four scenarios and a user study.

I. INTRODUCTION

  • Robots serving human-centered roles must interact with people and navigate public spaces with pedestrians, making socially compliant behavior important.The cited roles include package and food delivery, service, and home assistance.Source: §I · p. 1
  • Contextually appropriate navigation is difficult because human behavior and environmental or task contexts are not easily categorized.Learning-based approaches require substantial datasets for training and reward engineering, and are hard to generalize.Source: §I · p. 1
  • Language models offer contextual understanding but are not well explored for social navigation and can have high latency that impedes smooth, efficient interaction.The paper identifies contextual appropriateness and politeness as requiring understanding of task and environmental context and human behavior.Source: §I · p. 1
  • VLM-Social-Nav uses a perception model to detect key social entities and prompt a VLM to compute social cost for a bottom-level planner.The approach is presented as navigating social scenarios using common sense without dedicated training on a large dataset.Source: §I · p. 2
  • At least 27.38% improvement in average success rate and 19.05% improvement in average collision rate were reported across four social navigation scenarios.The user study score also showed the most socially compliant navigation behavior.Source: §I · p. 2

II. RELATED WORK

Prior social-navigation work addresses safety and contextual appropriateness through planning and learning, but data needs and context dependence remain challenges. Foundation models have navigation applications, yet their use for social navigation is described as underexplored.

II. RELATED WORK

  • The related-work overview frames safety requirements, contextual appropriateness, and foundation models as areas relevant to social robot navigation.The system architecture detects social entities and uses VLM scoring to compute social cost for robot actions.Source: §II · p. 2, Figure 2 · p. 3

A. Safety Requirement of Social Navigation

  • DWA selects feasible actions using collision constraints, while VO-based methods and learning-based approaches address other navigation challenges with distinct limits.VO methods do not account for uncertainty; PRVO and OFVO require hard planning thresholds, while learning-based approaches need substantial data or realistic simulators.Source: A. Safety Requirement of Social… · p. 2

B. Contextual Appropriateness of Social Navigation

  • Social compliance beyond safety depends on scenario and culture, and contextual appropriateness remains a challenge for social navigation.SCAND and MuSoHu are large-scale datasets collected for learning socially compliant navigation behaviors.Source: B. Contextual Appropriateness of… · p. 2

C. Large Foundation Models for Navigation

  • Foundation models have been used for task planning, semantic mapping, and language-guided navigation, but are not extensively investigated for social navigation.The related work identifies contextual understanding and commonsense reasoning as capabilities of language models.Source: C. Large Foundation Models for… · p. 2

III. APPROACH

The approach defines social robot navigation as cost-minimizing motion planning that treats people as social entities. Its social cost measures deviation from behavior a person would adopt under social conventions, with a VLM used to estimate that behavior.

A. Problem Definition

  • Social robot navigation is modeled as an MDP in which a planner selects linear and angular velocities to minimize expected cost.The state is robot pose, and the cost function maps states and actions to a cost.Source: A. Problem Definition · p. 3
  • The overall cost combines goal-seeking, obstacle-avoidance, and social-norm terms, each weighted by a non-negative coefficient.The weights are α, β, and γ, respectively.Source: A. Problem Definition · p. 3
  • Social cost is defined as deviation between robot behavior and behavior a human would adopt under social conventions.Minimizing this deviation encourages the robot to emulate socially acceptable human behavior.Source: A. Problem Definition · p. 3
  • A VLM estimates the convention-appropriate human behavior from perceived images and prompts, avoiding the need to derive every social factor mathematically.The paper notes that such factors are challenging to define mathematically and that human behavior can alternatively be obtained from large datasets.Source: A. Problem Definition · p. 3

B. VLM-based Social Navigation Architecture

VLM-Social-Nav combines perception, a VLM-based scoring module, and an optimization-based planner. The VLM estimates a socially appropriate behavior from observations and instructions; the module converts it into social cost for the planner, and perception gates VLM use to detected social cues.

B. VLM-based Social Navigation Architecture

  • The architecture integrates a perception layer with an optimization-based motion planner that selects actions by minimizing the system cost.LiDAR supplies geometric information for obstacle avoidance, while RGB images provide contextual details for social navigation.Source: B. VLM-based Social Navigation… · p. 3
  • Perception detects social entities such as people, gestures, and doors, and activates the social cost only when such cues are detected.Gating the module reduces VLM queries to support real-time navigation efficiency.Source: B. VLM-based Social Navigation… · p. 3, C. VLM-based Scoring Module · p. 4

C. VLM-based Scoring Module

  • The scoring module translates robot observations and textual instructions into a social cost used by the low-level planner to choose an appropriate action.The planner minimizes the combined goal, obstacle, and social costs.Source: C. VLM-based Scoring Module · p. 4
  • The VLM infers an immediate socially compatible behavior from the robot-view image, prompt, and current action.The prompt uses zero-shot examples to leverage VLM reasoning without extensive training.Source: C. VLM-based Scoring Module · pp. 3–4
  • The VLM response specifies heading and speed, which the scoring module uses to calculate the next-step social cost.The prompt provides the current action and maps angular velocity to left, straight, or right instructions; it also includes walking-etiquette guidance.Source: C. VLM-based Scoring Module · p. 4, Figure 3 · p. 4
  • Prompt reasoning guidelines can adapt navigation to local rules, such as instructing the robot to pass on the left where that is customary.The paper says such guidelines enhance decision-making and adaptation to specific rules.Source: C. VLM-based Scoring Module · p. 4

IV. EXPERIMENTS

The section introduces the implementation and experimental results covered next.

IV. EXPERIMENTS

  • The section describes implementation details and qualitative and quantitative experimental results.Source: §IV · p. 4

A. Implementation Details

The implementation uses a Turtlebot 2 with onboard sensing and computing, YOLO perception, GPT-4V, and a DWA planner. Evaluation follows established social-navigation scenarios and compares against DWA and a dataset-trained behavior-cloning baseline.

A. Implementation Details

  • The Turtlebot 2 uses a Velodyne VLP16 LiDAR, Zed 2i camera, Intel i7 CPU, and Nvidia GeForce RTX 2080 GPU.YOLO detects key objects, with humans, doors, and gestures emphasized as social cues.Source: A. Implementation Details · p. 4
  • GPT-4V interprets social dynamics and proposes an immediate robot action, while the low-level planner is DWA.The authors report that GPT-4V produced reliable results with high consistency and an average inference time of around three seconds in a preliminary comparison.Source: A. Implementation Details · p. 4
  • The comparisons include DWA without social cost and behavior cloning trained on SCAND, whose demonstrations include behaviors such as keeping right and yielding.The evaluation follows prior social-navigation studies and includes qualitative, quantitative, and user-study results across four scenarios.Source: A. Implementation Details · pp. 4–5
  • The four scenarios are frontal approach, frontal approach with a stop gesture, perpendicular-path intersection, and crossing through a narrow doorway.In the doorway scenario, the robot is expected to wait outside and yield to the human.Source: A. Implementation Details · p. 5

B. Qualitative Result

The scenarios define expected yielding and path choices around people, while observed trajectories show VLM-Social-Nav following these conventions more closely than the compared methods. Its behavior includes yielding to gestures and passing behind people at intersections.

B. Qualitative Result

  • In frontal approaches, the robot is expected to yield or slow down and modify its trajectory to avoid obstructing the person, conventionally keeping right.With a gesture, it is expected to yield by interpreting the human gesture.Source: B. Qualitative Result · p. 6
  • At intersections, the robot is expected to approach slowly and may stop or pass behind the person to avoid obstructing their path.Source: B. Qualitative Result · p. 6
  • VLM-Social-Nav predominantly passes on the right in frontal approaches and passes behind people at intersections, aligning its trajectories more closely with social norms than DWA.DWA also avoids collisions but may pass on either side or veer in front of a person; BC avoids people but often fails to recover its original path and reach the goal.Source: B. Qualitative Result · p. 6
  • At narrow doorways, the robot is expected to wait outside and yield to the human.Source: B. Qualitative Result · p. 6

C. Quantitative Result

Across four scenarios, VLM-Social-Nav improves average success and collision rates, and it is the only method reported to respond successfully to participants’ gestures. The user-study figure reports higher agreement scores for VLM-Social-Nav across questions.

C. Quantitative Result

  • Success rate measures goal attainment, while collision rate includes collisions with people or objects and manual interventions to prevent imminent collisions.The user-study score is the average score from the study described in Section IV-D.Source: C. Quantitative Result · p. 6
  • The user-study figure reports per-question average scores for three methods in each scenario, with VLM-Social-Nav receiving the highest participant agreement across questions.Source: Figure 5 · p. 7
  • 27.38% improvement in average success rate and 19.05% reduction in average collision rate across four social navigation scenarios.The results are averaged over 21 runs per method and scenario; only VLM-Social-Nav successfully responded to participants’ gestures.Source: C. Quantitative Result · pp. 6–7

D. User Study

The user study evaluates participants’ agreement with statements about robot motion across the social-navigation scenarios. VLM-Social-Nav receives the highest agreement across questions, while BC scores vary and DWA’s score is closer in the narrow doorway scenario.

D. User Study

  • Participants walk along a predefined trajectory and rate agreement with robot-motion statements on a five-level Likert scale.The three methods are randomly shuffled and repeated three times, and each scenario is tested with seven participants.Source: D. User Study · p. 7
  • VLM-Social-Nav receives the highest participant agreement across questions, whereas the BC method has large standard error, indicating inconsistent performance.The score difference between VLM-Social-Nav and DWA is not large in the narrow doorway, where DWA often freezes after failing to find a plan, resembling VLM-Social-Nav’s complete stop.Source: D. User Study · p. 7

E. Discussion

The discussion addresses VLM response latency and the limits of the tested settings. Prompt choices and cue-triggered queries yield average response times of 2–3 seconds, but frequent interactions may challenge this approach.

E. Discussion

  • Average VLM response times of 2–3 seconds are achieved through formatted prompts, predefined choices, and querying only when detected social cues make it necessary.Frequent interactions in more dynamic scenarios may make this latency problematic.Source: E. Discussion · p. 7
  • VLMs can reason about social interactions from single images, but their mistakes make using their outputs alone unsafe; the approach instead incorporates them as a cost term.Source: E. Discussion · p. 7
  • Robot experiments were conducted only indoors, and extending the approach to outdoor global navigation remains future work.The authors also aim to extend the approach to multiple individuals; simple left-or-right directions may not suffice when multiple groups are present.Source: E. Discussion · p. 7

V. CONCLUSION

VLM-Social-Nav uses perception and VLM-based scoring to guide socially compliant robot behavior, reducing reliance on extensive training data and explicit rules. The system was evaluated in four real-world Turtlebot scenarios, with several directions proposed for future work.

V. CONCLUSION

  • VLM-Social-Nav detects social entities and uses VLM-generated guidance and scoring to support real-time, socially compliant robot actions.The approach reduces dependence on extensive training data and removes the need for explicit rules or hand-tuned parameters typically associated with imitation learning.Source: §V · p. 7
  • The system was evaluated in four real-world social navigation scenarios using a Turtlebot robot.The paper describes VLMs as interpreting social situations and deriving actions from social rules, a more nuanced approach than simpler rule- or planning-based methods.Source: §V · p. 8
  • Future work includes visual prompting for spatial reasoning, exploring open-source VLMs to help address hallucinations, and integrating chain-of-thought-generated social instructions into autonomous navigation.The authors also propose using lower-level information such as log probabilities to detect and address hallucinations.Source: §V · p. 8

Related papers

Loading 2404.00210v3…