Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigation

• Social navigation involves recognizing and reacting to other agents’ objectives while adjusting the robot’s path and projecting signals that help reciprocal interaction.Source: §I · p. 1
• Demonstration data can support Learning from Demonstrations and help researchers understand human navigation around autonomous robots.Source: §I · p. 1
• Existing social-navigation datasets contain limited interactions in constrained settings or focus exclusively on indoor navigation, omitting naturally occurring behaviors.Source: §I · p. 1
• Imitation learning avoids explicitly defining complex social-navigation rules, yet large-scale datasets of socially compliant demonstrations in the wild remain unavailable.Source: §I · pp. 1–2
• 8.7 hours, 138 trajectories, and 25 miles of multimodal demonstrations were collected from two morphologically different robots, including lidar, joystick, odometry, visual, and inertial data.Source: §I · p. 2

• Learning-based robot-navigation methods address adaptive planner parameters, viewpoint invariance in demonstrations, and end-to-end autonomous driving.Source: A. Learning for Robot Navigation · p. 2
• Tai et al. use simulation and the social force model to generate demonstrations, then train a social-navigation policy with Generative Adversarial Imitation Learning.Source: A. Learning for Robot Navigation · p. 2
• Reinforcement-learning approaches have shown real-world results but remain limited to specific social scenarios and require simulated episodic learning.Source: A. Learning for Robot Navigation · p. 2
• Inverse reinforcement learning has been used to learn cost functions for socially compliant navigation from human demonstrations.Source: A. Learning for Robot Navigation · p. 2

• Simulated social environments enable fast data collection for social navigation.Source: 1) Simulated Datasets for Social… · p. 2
• Simulation can specify the number and locations of humans, room structure, objects, and interactions between people and objects.Source: 1) Simulated Datasets for Social… · p. 2
• Simulated platforms are limited because they lack natural real-world interactions.Source: 1) Simulated Datasets for Social… · p. 2

2) Real-world Datasets for Robot Navigation:

• Prior real-world datasets include laser scans, odometry, localization, LiDAR, RGBD, GPS, and IMU data for long-term robot navigation and perception.Source: 2) Real-world Datasets for Robot… · p. 3
• Teleoperated demonstrations in prior datasets are not explicitly socially compliant, limiting their use for studying socially compliant navigation strategies.Source: 2) Real-world Datasets for Robot… · p. 3
• JRDB contains 64 minutes from 54 indoor and outdoor trajectories and focuses on perception tasks, whereas SCAND focuses on the navigation component of social navigation.Source: 2) Real-world Datasets for Robot… · p. 3
• SCAND provides joystick commands, multimodal sensor data, and labels for twelve social interactions across naturally occurring scenarios.Source: 2) Real-world Datasets for Robot… · p. 3
• Data from the legged Spot and wheeled Jackal supports investigation of how robot morphology affects navigation and induced social interactions.Source: 2) Real-world Datasets for Robot… · p. 3

Collection setting

• Four human demonstrators teleoperated Jackal and Spot robots across 138 trajectories on the University of Texas at Austin campus.Data collection included sidewalks, roads, lawns, and building interiors with people present during high-traffic periods.Source: A. Collecting Data · pp. 3–4

Collection procedure

• The human operator followed behind each robot at an average distance of two meters while teleoperating with a joystick.This procedure was used for every trajectory in SCAND.Source: A. Collecting Data · p. 3

Social scenarios

• The demonstrations cover naturally occurring social navigation scenarios, including street crossings, narrow doorways, crowds, vehicle interactions, and stationary queues.Figure 2 pairs RGB imagery with lidar and Spot side-camera views for five example scenarios.Source: Figure 2 · p. 4, A. Collecting Data · p. 4

Sensor suite

• SCAND records Velodyne lidar, joystick commands, odometry, camera visuals, and 6D inertial information from both robots.Jackal additionally provides stereo imagery and wheel odometry, while Spot provides five body-mounted monocular cameras and visual odometry.Source: A. Collecting Data · p. 4

Annotation scope

• SCAND provides navigation-relevant multimodal data but no labeled annotations for human detection or tracking.The dataset instead includes demonstrator velocity commands and labels for 12 social interactions in every trajectory.Source: A. Collecting Data · p. 4

Labeled Annotations of Social Interactions

• Each SCAND trajectory is annotated with textual captions selected from twelve predefined social-interaction labels.The labels describe interactions occurring along the path and are intended to support studies of specific real-world navigation scenarios.Source: B. Labeled Annotations of Social… · p. 4
• The analysis asks whether socially compliant navigation has multiple strategies and whether SCAND can support jointly learned global and local planners.These questions are addressed through demonstrator classification and behavior cloning.Source: §IV · pp. 4–5
• The dataset’s multimodal information includes lidar, monocular and RGB imagery, inertial sensing, and joystick commands issued during demonstrations.Figure 3 presents sensors on the Jackal and Spot robots, while joystick commands accompany the sensor streams.Source: Figure 3 · p. 5, §IV · p. 5
• The demonstrator classifier uses ten-second sensor-observation sequences to predict the identity of the human demonstrator.The behavior-cloning agent uses a related architecture with global- and local-planner heads instead of a classifier head.Source: Figure 4 · p. 5, §IV · p. 5

A. Demonstrator Classification

• The authors hypothesize that multiple socially compliant navigation strategies can exist in the same scenario.The question concerns whether different strategies are possible for navigating socially in an environment.Source: A. Demonstrator Classification · p. 5

Approach and Implementation

• Sixteen trajectories from two demonstrators on the same Speedway road route were used, with twelve for training and four for validation.The classifier was trained to identify the demonstrator from navigation data.Source: 1) Approach and Implementation · p. 5
• The classifier processes ten-second sequences containing lidar BEV images, relative robot positions, future trajectories, inertial values, and joystick commands.A convolutional encoder handles lidar images, while fully connected networks process other observations and produce the classification output.Source: 1) Approach and Implementation · p. 5

Results and Conclusion

• 74.48% accuracy was achieved when classifying the demonstrator on the held-out test set.The paper contrasts this with a 50% random-guessing success rate and interprets the result as evidence of distinguishable navigation styles.Source: 2) Results and Conclusion · p. 5

Approach and Implementation

• Behavior Cloning jointly trains end-to-end global and local planners from SCAND demonstrations.The global planner predicts a future socially compliant trajectory, while the local planner predicts demonstrated forward and angular velocities.Source: 1) Approach and Implementation · pp. 5–6

Results

• Fig. 5 contrasts demonstrated, move_base, and learned trajectories as a human crosses the robot’s path.The learned path closely follows the socially compliant demonstration, whereas move_base turns toward the human’s future state.Source: Figure 5 · p. 6, 2) Results and Conclusion · p. 6
• 0.26 average Hausdorff distance separates the learned global planner’s predicted trajectory from the demonstrated path, versus 1.25 for move_base.The evaluation uses a held-out test set and compares predicted or global paths with the demonstrator’s future path.Source: 2) Results and Conclusion · p. 6
• Fourteen participants evaluated static and dynamic human-robot scenarios for perceived social compliance and safety.The static scenario used a stationary human, while the dynamic scenario involved the robot and human moving toward each other’s starting positions.Source: 2) Results and Conclusion · p. 6, Figure 6 · p. 6
• SCAND’s policy scored higher than move_base for social compliance and safety in the human evaluation.Mean social-compliance scores were 4.39 versus 2.86, and safety scores were 4.71 versus 2.89; both comparisons were statistically significant by one-way ANOVA.Source: 2) Results and Conclusion · p. 6, Figure 7 · p. 7

Limitations

• The BC agent handles simple social-navigation scenarios, but more sophisticated scenarios in SCAND may require better imitation-learning algorithms.This is identified as a limitation of the demonstrated validation scope.Source: 2) Results and Conclusion · p. 6

Generalization

• Infrequent novel interactions may limit generalizability to unseen situations.The paper proposes representation learning with SCAND as one promising direction for improving generalization.Source: §V · p. 7
• Collection in Austin may encode regional norms such as staying right or overtaking pedestrians from the left.The paper calls for algorithms, evaluations, and metrics flexible enough to accommodate different local norms.Source: §V · p. 7

Benchmarking and Transfer

• SCAND could augment simulation benchmarks with human-robot interaction trajectories and support real-to-sim transfer.The paper identifies both directions as future work because benchmarking is outside its scope.Source: §V · p. 7

Learning Applications

• SCAND enables future trajectory prediction, trajectory classification, and inverse reinforcement learning for large-scale socially compliant cost-function learning.These directions extend earlier work based on human-only or robot-only trajectories.Source: §V · p. 7

Conclusion

• SCAND contains 8.7 hours, 138 trajectories, and 25 miles of demonstrations collected on two morphologically different robots.It also includes multimodal sensory streams and labeled annotations of social interactions.Source: §VI · p. 7
• A neural-network demonstrator-classification task shows that SCAND contains more than one socially compliant navigation strategy.The conclusion presents this as the first of several questions addressed with the dataset.Source: §VI · p. 7
• Behavior cloning on SCAND learns socially compliant global and local planners for mobile-robot navigation.The conclusion reports this as evidence that the dataset supports learning both planning levels.Source: §VI · p. 7
• Human trials on two social-navigation scenarios find the behavior-cloned local planner relatively more socially compliant and safe.The conclusion summarizes the validation of the local planner through human evaluation.Source: §VI · p. 7