PaperlayerView exact revision on arXiv

Source-linked AI summary

Principles and Guidelines for Evaluating Social Robot Navigation Algorithms

Anthony Francis, Claudia Pérez-D'Arpino, Chengshu Li, Fei Xia, Alexandre Alahi, Rachid Alami, Aniket Bera, Abhijat Biswas, Joydeep Biswas, Rohan Chandra, Hao-Tien Lewis Chiang, Michael Everett, Sehoon Ha, Justin Hart, Jonathan P. How, Haresh Karnan, Tsang-Wei Edward Lee, Luis J. Manso, Reuth Mirksy, Sören Pirk, Phani Teja Singamaneni, Peter Stone, Ada V. Taylor, Peter Trautman, Nathan Tsoi, Marynel Vázquez, Xuesu Xiao, Peng Xu, Naoki Yokoyama, Alexander Toshev, Roberto Martín-Martín

arXiv:2306.16740v4cs.ROcs.AIcs.HCcs.LG

TL;DR

Fair evaluation of social robot navigation is difficult because it must account for dynamic human agents and subjective judgments of robot behavior. The paper defines socially navigating robots and proposes benchmarking criteria, evaluation guidelines, and a metrics framework for comparing results across settings.

  • Problem

    Social navigation lacks consensus metrics because evaluation must capture multiple encounter aspects, including safety and communication of intent, often from human perspectives.

  • Method

    The paper reviews social navigation methodology, defines taxonomies and criteria for metrics, scenarios, benchmarks, datasets, and simulators, and outlines a research lifecycle for collecting and evaluating data.

  • Results

    The paper defines a socially navigating robot through eight principles and proposes criteria for benchmarks that evaluate social behavior with quantitative, human-grounded, efficient, repeatable, scalable, and validated methods.

  • Takeaways & Limitations

    Common benchmarking criteria can support fairer comparison of social navigation methods across simulators, robots, datasets, and evaluation settings.

  • Takeaways & Limitations

    Subjective metrics and realistic real-world evaluations remain difficult to scale, while existing benchmarks trade off scalability against grounding in human data or manual setup.

Abstract

from arXiv · show

A major challenge to deploying robots widely is navigation in human-populated environments, commonly referred to as social robot navigation. While the field of social navigation has advanced tremendously in recent years, the fair evaluation of algorithms that tackle social navigation remains hard because it involves not just robotic agents moving in static environments but also dynamic human agents and their perceptions of the appropriateness of robot behavior. In contrast, clear, repeatable, and accessible benchmarks have accelerated progress in fields like computer vision, natural language processing and traditional robot navigation by enabling researchers to fairly compare algorithms, revealing limitations of existing solutions and illuminating promising new directions. We believe the same approach can benefit social navigation. In this paper, we pave the road towards common, widely accessible, and repeatable benchmarking criteria to evaluate social robot navigation. Our contributions include (a) a definition of a socially navigating robot as one that respects the principles of safety, comfort, legibility, politeness, social competency, agent understanding, proactivity, and responsiveness to context, (b) guidelines for the use of metrics, development of scenarios, benchmarks, datasets, and simulators to evaluate social navigation, and (c) a design of a social navigation metrics framework to make it easier to compare results from different simulators, robots and datasets.

I. INTRODUCTION

Social robot navigation lacks a shared definition and comparable evaluation practices. The paper proposes a taxonomy, eight principles, evaluation guidelines, and a common API for more comparable research.

  • The field lacks consensus on what social navigation means or how robots should achieve it.
  • The paper develops a taxonomy and recommendations for evaluating metrics, scenarios, benchmarks, datasets, and simulators.
  • The guidelines distinguish high-level principles from concrete recommendations for creating and testing social navigation solutions.
  • A socially navigating robot achieves its navigation goals while modifying behavior so surrounding agents’ experiences are not degraded or are enhanced.
  • Eight principles guide social navigation: safety, comfort, legibility, politeness, social competency, agent understanding, proactivity, and contextual appropriateness.
  • Contextual appropriateness means evaluating behavior within cultural, diversity, environmental, operational, task, and interpersonal contexts.

IV. RESEARCH METHODOLOGIES OF SOCIAL NAVIGATION

Social navigation evaluation spans human perceptions and algorithmic responses to dynamic obstacles, producing varied studies and a research lifecycle rather than one uniform methodology.

  • Benchmark methodologies address both human perceptions of robot behavior and algorithms’ responses to dynamic obstacles.
  • Different scientific questions require different data-gathering studies, which guide method development and form a social navigation research lifecycle.

A. Research Questions of Social Navigation

Social navigation research asks how methods compare, how components contribute, how behaviors generalize, and how humans react. The paper organizes these questions across complementary field, deployment, staged, laboratory, and benchmark studies.

  • Research Questions of Social Navigation: Research questions include comparing methods against baselines, evaluating component effects, testing generalization, rating socialness, analyzing behavior change, and discovering issues.
  • Research Questions of Social Navigation: Quantitative simulation metrics can assess issues such as safety under increasing obstacle density, while subjective principles often require human evaluation.
  • Research Questions of Social Navigation: Human ratings require proper HRI protocols and validated survey instruments, while behavior-analysis studies require blinding for valid participant and rater reactions.
  • Research Questions of Social Navigation: A comprehensive evaluation methodology does not yet exist because different researchers and groups have different aims and needs.
  • Research Questions of Social Navigation: The paper therefore proposes a methodology for principled decisions rather than one evaluation protocol for all social navigation methods.
  • Types of Social Navigation Studies: Field studies collect natural but nonreproducible interactions, whereas deployments and staged interactions provide more experimental control while potentially changing participant responses.
  • Types of Social Navigation Studies: Validated scenarios enable analysis of known problems but may miss novel conditions or problems appearing in the wild.
  • Lifecycle of Social Navigation Research: The proposed lifecycle uses data collection, issue discovery, experiments and scenarios, benchmarks, and public challenges to connect research stages.

V. A TAXONOMY OF SOCIAL NAVIGATION

The paper develops a common taxonomy for analyzing social navigation research instruments across metrics, scenarios, benchmarks, datasets, and simulators. It organizes these instruments around shared factors while recognizing instrument-specific dimensions.

  • Taxonomy scope: The taxonomy analyzes metrics, datasets, simulators, scenarios, and benchmarks using a common vocabulary of formal factors.These instruments are treated as research tools whose characteristics can be compared systematically.
  • Instrument types: Benchmarks use evaluation protocols to collect metrics, while publicly available challenges add success criteria, evaluation mechanisms, and leaderboards.The paper distinguishes general benchmarks from challenges designed to compare solutions publicly.
  • Instrument types: Datasets require analysis of coverage, sampling distribution, annotations, privacy, and fairness, whereas simulators require attention to differing APIs and metrics.The authors are working toward a common simulator API to improve comparisons across simulation environments.
  • Cross-instrument comparison: Social navigation benchmarks, datasets, and simulators share challenges in characterizing contexts, environments, robot roles, tasks, and embodiments.The authors therefore present common factors instead of analyzing these instruments entirely separately.
  • Common factors: Shared factors include context, physical environment, human user type, human behavior, robot task, and robot role.These factors describe the setting, participants, behaviors, and intended relationship between robots and humans.
  • Common factors: Scenarios combine physical environment, human behavior, and robot task into configurations that can be scripted or unscripted.Robot role may also be specified as part of a scenario.

VI. SOCIAL NAVIGATION METRICS

Social navigation metrics are difficult to standardize because they assess multiple dimensions of human-robot encounters, including safety and communication of intent. The paper organizes metrics by nature, modeled variables, and temporal scope, while contrasting human-surveyed measures with algorithmic proxies.

  • Motivation: Social navigation lacks a single consensus metric because evaluation must address multiple aspects of human-robot encounters.Examples include safety near people and how well robots communicate their intent for motion coordination.
  • Motivation: Safety itself can refer to physical collisions, psychological comfort, or avoiding disruption of social activity.The paper uses safety to illustrate why social navigation concepts can have multiple dimensions.
  • Metric taxonomy: Metrics are classified by their nature, the variables they model, and their temporal scope.The paper recommends applying all three taxonomies to fully classify a metric.
  • Metric nature: Surveyed metrics capture human ratings through questionnaires or sensors but are expensive, difficult to scale, time-consuming, and potentially high-variance or non-reproducible.Questionnaires explicitly request ratings, whereas sensors transduce ratings from sensor data.
  • Metric nature: Algorithmic metrics are cheap and reproducible proxies for surveyed metrics, but social navigation comparisons typically require multiple metrics rather than one reference measure.Algorithmic metrics may be hand-crafted or learned from data gathered from surveys.

2) Taxonomy based on the variable being modeled:

Social-navigation evaluation metrics differ in the phenomena they model, their temporal scope, and their suitability for benchmarking or diagnosis. Reliable evaluation must also account for dynamic environments, changing human behavior, subjective and difficult-to-scale measurements, context dependence, and the absence of a universally superior metric.

  • Variable modeled: Metrics may assess non-social performance, social phenomena, or both, with non-social metrics offering objective and reproducible measures but no social-performance information.Non-social metrics include path and energy efficiency or success rate; social metrics include comfort, acceptance, trustworthiness, and predictability.
  • Temporal scope: Task-wise metrics provide one score per task and are generally more appropriate for benchmarking, whereas step-wise metrics provide scores over time and suit path planning.Step-wise data can be aggregated into task-wise measures, but temporal weighting may be needed because moments in a task can differ in social importance.
  • Evaluation challenges: Evaluation must account for dynamic environments, long-term participant exposure, deployment effects, and limitations of the metrics themselves.Lighting, weather, and battery state can affect reported accuracy or speed, while human behavior changes through repeated interaction with robots.
  • Metric limitations: Human ratings are subjective and subjective metrics are difficult to scale, while realistic real-world evaluations are also difficult to reproduce in laboratory settings.Participant culture, context, goals, robot familiarity, scenario complexity, and the disruption caused by frequent feedback can affect measurement.
  • Metric selection: Metric value is highly context-dependent, and no convincing quantitative method establishes that one social-navigation metric is strictly better than another.Survey-based metrics are generally preferred for benchmarking but are resource-intensive and difficult to reproduce when run incorrectly; specific metrics remain useful for diagnosing algorithmic flaws.

D. Recommendations for Metric Usage and Development

The paper recommends combining standard navigation, hand-crafted social, learned, and validated survey metrics, with context-aware parameters and detailed distributions to support fair comparison.

  • Metric and survey choices: Surveyed metrics can be meaningful and reliable but are expensive, time-consuming, difficult to design, and sometimes unsuitable for ablation studies.The paper recommends iterative questionnaire validation because no unified social-navigation questionnaire approach exists.
  • Metric and survey choices: A common subset of hand-crafted metrics should cover success, collisions, failures, trajectory properties, and social aspects.Table I lists the phenomena, parameters, units, ranges, and references associated with these metrics.
  • Context and reporting: Metric values depend heavily on task, context, and parameter settings, so reports should state the relevant parameters and experimental context.This dependence means identical metric values may not represent equivalent performance across settings.
  • Context and reporting: Results across multiple trajectories should include distributions alongside averages, because outliers and consistency matter especially for safety.Pointwise estimates of single metrics can distort system-performance judgments.
  • Development sequence: Validated human surveys should evaluate social performance after policies become sufficiently robust, while algorithmic metrics can filter policies in simulation.The recommended development sequence also uses learned metrics where available to iterate on behavior or support acceptance tests.
  • Development sequence: Experiments should use consistent setups, iterative analysis, and broad metric batteries to reduce bias and improve comparability.Relevant distortions include environmental complexity, subject selection, robot familiarity, survey fatigue, and differing experimental setups.

A. Scenario Design Methodology

The scenario-design methodology defines interactions precisely enough for identification and experimentation while remaining flexible enough to capture natural variation and long-tail behavior.

  • Scenario reuse: Scenario development should bridge field studies, robot deployments, laboratory experiments, and dataset-generation or imitation-learning use cases.The FRONTAL APPROACH scenario illustrates how one interaction definition can be reused across these settings.
  • Define the scenario: Scenario design begins by specifying research context, intended robot task, intended human behavior, and success metrics.The robot task should be unambiguous enough to determine whether it succeeded.
  • Evaluate the definition: Scenarios should be common enough to matter, flexible enough to capture wild behavior, and fit for evaluating the intended research question.Overly narrow scenarios can prescribe behavior and exclude naturally occurring variants.
  • Communicate scenarios: A Social Navigation Scenario Card is proposed to communicate scenario definitions consistently after scenario-development and evaluation.The paper presents the card as a step toward more formal scenario engineering.

B. Social Navigation Scenario Cards

Scenario Cards organize reusable social-navigation scenarios into metadata, a broad operational definition, and a usage guide covering evaluation and experimental details.

  • Scenario reuse: Cards are intended to support scenarios ranging from simple pedestrian encounters to complex crowd navigation while remaining reusable.The paper gives FRONTAL APPROACH as a common example and notes that cards should remain flexible across use cases.
  • Card structure: Each Scenario Card contains metadata, a Scenario Definition, and a Scenario Usage Guide.These elements identify the scenario, describe its environment and agents, and specify specialized evaluation information.
  • Scenario metadata: Scenario metadata records an unambiguous name, description, type, and scientific purpose or research context.Contexts can distinguish location, pedestrian density, and high-level task.
  • Scenario definition: The Scenario Definition specifies environment, robot task, and human behavior precisely enough for labeling but broadly enough to preserve behavioral diversity.FRONTAL APPROACH should include variants such as a human stopping or changing direction.
  • Scenario usage: The Usage Guide adds labeling criteria, success and quality measures, ideal outcomes, failure modes, human playbooks, and contextual variants.Context can alter success metrics, ideal outcomes, failure modes, and expected human behavior.

C. Example Social Navigation Scenarios

The paper recommends evaluating policies across common, context-appropriate scenarios and communicating them with standardized cards, while benchmarks combine scenarios and metrics for method comparison.

  • Scenario coverage: Policies should be tested on scenarios covering their intended use cases, such as hallways and doorways for pedestrians or traffic-flow cases for crowds.A suitable standard benchmark may not exist for every new research purpose.
  • Scenario taxonomy: Example social-navigation scenarios include pedestrian, crowd, interaction, approach, hallway, intersection, and traffic-flow cases.Table III groups scenarios by scientific purpose and related scenario types.
  • Scenario illustrations: Figure 7 depicts robot and human positions, motion directions, intended paths and destinations, obstructions, and agent-emitted signals for example scenarios.The visual encoding distinguishes robot elements in blue, human elements in red, obstructions in grey, and signals in green.
  • Scenario guidelines: New scenarios should specify context, robot task, human behavior, success metrics, flexibility, and fitness for purpose.These guidelines are organized as N1 through N7 before the communication recommendation N8.
  • Scenario guidelines: Scenario Cards should be used as a standard format for communicating new scenarios or specialized versions of existing scenarios.The format is intended to make scenario content clear to other researchers.
  • Benchmarking: Benchmarks improve on isolated experiments by collecting scenarios into suites with specified metrics, enabling comparisons among methods.The paper analyzes protocols, environments, and challenges and recommends quantitative social measures and validated questionnaires grounded in human data.

A. Expanding the Factors for Benchmark Analysis

Benchmark analysis should consider platforms, datasets, baselines, reproducibility, hardware, and human-behavior authoring alongside scenarios and metrics. Existing benchmarks span abstract dynamic-obstacle tests, simulated human interactions, and physical-experiment protocols, with differing capabilities and limitations.

  • Benchmark analysis factors: Benchmarks should specify a simulation platform, associated datasets, provided baselines, challenge leaderboards, downloadability, update recency, hardware support, and human-behavior authoring methods.These factors determine how readily benchmarks can be reproduced, compared, and adapted across robot platforms and behavioral settings.
  • Benchmark analysis factors: Social navigation benchmarks combine a system that runs algorithms and pedestrians, well-defined scenarios, and evaluation metrics, with datasets optionally included.The benchmark definition distinguishes these integrated evaluations from datasets that only contain reference behavior.
  • Benchmarking protocols: The SOCIAL NAVIGATION PROTOCOL specifies scenarios and expected human-robot interaction outcomes to collect expert trajectories and evaluate policies with low variability.Its scenarios include Frontal Approach, Blind Corner, and Corridor Intersection, but each experiment requires manual setup rather than a downloadable simulated environment.
  • Benchmark scope: Existing benchmarks differ in support for human evaluation, baselines, pedestrian models, metrics, robot morphologies, and simulation environments.ARENABENCH, DYNABARN, GYM-COLLISION-AVOIDANCE, HUNAVSIM, SOCNAVBENCH, CROWDBOT, and SEANAVBENCH illustrate this variation.
  • Benchmark scope: Common benchmarks cover dynamic obstacle avoidance, simulated interactions with moving humans, crowd navigation, and physical-experiment protocols.Examples include downloadable simulators, crowd challenges, and protocols with varying human-behavior fidelity.

C. Strengths and Limitations of Existing Benchmarks

Existing benchmarks support diverse social-navigation scopes and often provide efficient simulated evaluation with metrics and sometimes baselines. However, benchmark types exhibit recurring trade-offs involving scalability, human grounding, setup requirements, and edge-case coverage.

  • Strengths: Benchmarks range from dynamic obstacle avoidance to human-robot interactions and crowd navigation, often offering downloadable simulation, metrics, and sometimes baselines.Their purposes include testing crowds, smaller social scenarios, algorithm improvements, and benchmark fidelity.
  • Limitations: Scalable benchmarks often lack human-data grounding, human-data benchmarks often require manual setup or extra components, protocols focus on human evaluations, and few cover meaningful edge cases.These are characterized as recurring limitations across benchmark types rather than as properties of every individual benchmark.
  • Guidelines: Recommended benchmark criteria are social-behavior evaluation, quantitative metrics, baselines, efficiency, repeatability, scalability, human-data grounding, and validated evaluation instruments.The criteria are intended to make benchmark results communicate broadly across the social-navigation community.
  • Guidelines: Good benchmarks should be efficient, repeatable, and scalable, with at least 30 samples suggested as a rule of thumb for real-robot trials.The paper notes that sample counts can instead be determined statistically when means and variances are available.
  • Guidelines: The paper recommends more human evaluation, standardized social questionnaires and quantitative metrics, and corner-case testing on standard navigation benchmarks.It identifies SEAN-EP, SOCIAL NAVIGATION PROTOCOL questionnaires, converging social metrics, BARN, and BENCH-MR as relevant resources.

(h) WILDTRACK

The dataset review examines social-navigation data across robot, pedestrian, simulated, multimodal, and teleoperated sources. It emphasizes coverage, sampling, behavior-authoring choices, annotations, and privacy when assessing dataset usefulness.

  • Behavior authoring: Teleoperator-based datasets should specify visibility, positioning, interaction effects, and use of multiple teleoperators to encourage behavioral diversity.Explicit control instructions are important when desired robot behavior differs from ordinary human behavior.
  • Coverage and sampling: Dataset scope should be broad enough to present interesting challenges but constrained by available resources so each scenario is thoroughly sampled.Well-sampled scenarios reduce the likelihood that trained methods encounter out-of-distribution situations within the dataset scope.
  • Dataset analysis: Dataset analysis considers robot hardware, sensors, behavior-authoring methods, collected data, coverage, sampling distribution, annotations, privacy, and fairness.These factors supplement the broader characteristics used to evaluate social-navigation datasets.
  • Annotations: Social-navigation annotations lack established interaction taxonomies, so existing activity-recognition datasets can provide a starting point.Annotations may cover whole episodes or segments, and human tracks may include boxes, skeletons, or gaze.
  • Privacy and fairness: Datasets involving humans require early decisions about anonymization and compliance with privacy-protection regulations.Privacy is identified as an important concern in social-navigation dataset construction.
  • Dataset types: Datasets span bird’s-eye pedestrian trajectories, egocentric robot and human recordings, synthetic interactions, multimodal robot data, and teleoperated socially compliant demonstrations.Examples include ETH/UCY, CROWDBOT, SOCNAV1/2, DYNABARN, JRDB, SCAND, MUSOHU, LCAS, and SACSON.

C. Guidelines for Datasets

Dataset guidelines prioritize broad but resource-supported coverage, systematic sampling and annotation, robot-specific behavior data, diverse platforms, command recording, and early privacy consideration. Simulator guidance complements these principles by requiring human-robot interaction and describing fidelity, interoperability, and agent representations.

  • Dataset guidelines: Datasets should be as broad as possible while ensuring resources suffice to explore their defined scope thoroughly.The paired guidelines balance community usefulness against the practical limits of data collection.
  • Dataset guidelines: Each dataset scenario should be well-sampled to improve representativeness and reduce out-of-distribution situations for methods trained on the data.Sampling is tied directly to whether the dataset adequately covers its stated scope.
  • Dataset guidelines: When robot behavior is the target, datasets should record actual robots rather than relying only on pedestrian data.Different robot morphologies may elicit different human responses, motivating diverse robot platforms when feasible.
  • Dataset guidelines: Dataset creators should record teleoperation commands or policy actions alongside ordinary sensor data.These commands expose how robot behavior was generated and support more explicit modeling of demonstrations.
  • Dataset guidelines: Annotations should be collected systematically, with behavior-generation and labeling methods specified, while privacy issues are considered early.Early privacy consideration addresses legal, policy, and moral issues associated with human data collection.
  • Simulator guidelines: A social-navigation simulator must support at least two agents in a social encounter; simulators otherwise range from simplified crowds to detailed human and environmental models.The review compares abstraction, focus, platforms, agent and scene representations, physics, robot fidelity, pedestrian fidelity, reactivity, and interoperability.
  • Simulator guidelines: Simulator comparisons should account for pedestrian motion and visual fidelity, reactivity, and interoperability across interfaces such as OpenAI Gym and ROS.These dimensions distinguish prerecorded, reactive, behaviorally diverse, and visually detailed simulations.

B. Existing Social Navigation Simulators

Existing social navigation simulators span diverse representations, behaviors, platforms, and task focuses, making cross-simulator comparisons difficult. The paper therefore supports specialized simulators while advocating common interfaces and shared features.

  • The surveyed platforms vary from simplified 2D multi-agent environments to systems with detailed pedestrians, realistic environments, and full robot morphologies.
  • Simulator behavior models include recorded trajectories, SFM, ORCA, behavior trees, attitudes, and custom or modulated policies.
  • Social navigation simulators target different problem areas, including crowd simulation, dynamic obstacles, social navigation, and collision avoidance.
  • Differences in pedestrian representation, such as cylinders versus walking gait, can make algorithm results from different simulators not directly comparable.
  • A common simulator or API could provide shared features, support training and evaluation across approaches, and promote reuse across specialized simulators.

2) Common Platforms:

The paper proposes common platforms and metrics interfaces to make social navigation data comparable across simulators, robots, and datasets. The design standardizes inputs, metric computation, outputs, and downstream analysis while preserving simulator diversity.

  • Recorded pedestrian trajectories preserve real-world motion but lose some fidelity and cannot react realistically when a robot deviates from the original data.
  • Reactive pedestrian models such as SFM and ORCA let researchers examine how robot-policy changes affect task performance, although no human-motion model is perfect.
  • A high-level metrics API defines shared data requirements, ROS and OpenAI Gym implementations, common library code, and a common output format.
  • The API requires agent poses over time, robot observations and actions, resulting trajectories, and static and dynamic obstacle geometry.
  • A standardized output format supports consistent analysis and visualization across compatible robots, simulators, and datasets, including datasets that cannot be replayed.

2) Implementation of Social Navigation API:

The proposed social navigation API is paired with reusable implementations and simulator guidelines covering interoperability, benchmarking, dataset generation, human labeling, extensibility, fidelity, and validation. The broader framework defines socially navigating robots and criteria for stronger evaluation.

  • 2) Implementation of Social Navigation API:: The open-source reference implementation includes JSON schemas for inputs and outputs, ROS and Gym generators, and reusable C++ and Python metric libraries.
  • 2) Implementation of Social Navigation API:: Researchers using the API must provide bridge code translating robot, simulator, or dataset data into the required format.
  • Common Platforms:: Simulator guidelines recommend standardized APIs, standard metrics, extensibility, dataset generation, benchmark creation, and human labeling.
  • Common Platforms:: Further guidelines recommend common robot morphologies, detailed pedestrians, diverse behavior-authoring options, and periodic validation against intended usage.
  • XI. CONCLUSIONS: The paper defines a socially navigating robot as one that achieves its goals while modifying behavior so other agents can better achieve theirs.
  • XI. CONCLUSIONS: Good benchmarks should evaluate social behavior with quantitative metrics, baselines, efficiency, repeatability, scalability, human data, and validated evaluation instruments.
Loading 2306.16740v4…