PaperlayerView exact revision on arXiv

Source-linked AI summary

STARS: From Spatiotemporal Dynamics to Social Representations in Human-Robot Interaction

Nathan Tsoi, Michael J. Munje, Tejas Oberoi, Rishab Maheshwari, Pengen Zheng, Tanush Chauhan, Peter Stone, Joydeep Biswas

arXiv:2609.40245v2cs.ROcs.LG

TL;DR

Social navigation calls for joint reasoning about spatial dynamics and social context, yet grounded HRI data is scarce and fragmented. STARS learns reusable representations by encoding interaction scenarios as spatiotemporal graphs and training a self-supervised graph autoencoder. The learned representations support data-efficient downstream generalization and encode social semantics without directly optimizing the encoder for downstream tasks.

  • Problem

    Social navigation needs joint spatial and social reasoning, but grounded HRI data is scarce and fragmented across modalities and tasks, making large unstructured models prone to overfitting.

    Source: §1 · pp. 1–2

  • Method

    STARS encodes fixed-duration multi-agent interaction scenarios as spatiotemporal graphs and learns node-level representations through self-supervised graph autoencoding for lightweight downstream probes.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    STARS representations support robust downstream generalization with a fraction of labeled data, transfer across two evaluated datasets, and encode social semantics without directly optimizing the encoder for downstream tasks.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The results provide evidence that structural priors can help offset scarce grounded HRI data and offer a direction toward reusable, data-efficient social-navigation representations.

    Source: §6 · p. 9

    The evaluation assumes fixed-duration interaction windows and processed agent tracks, and is limited to curated social-navigation datasets.

    Source: §5 · p. 9

Abstract

from arXiv · show

Robot navigation in dynamic, human-centered environments requires socially-compliant decisions grounded in robust scene understanding. Recent Vision-Language Models (VLMs) exhibit promising capabilities such as object recognition, common-sense reasoning, and contextual understanding, capabilities that align with the nuanced requirements of social robot navigation. However, it remains unclear whether VLMs can accurately understand complex social navigation scenes (e.g., inferring the spatial-temporal relations among agents and human intentions), which is essential for safe and socially compliant robot navigation. While some recent works have explored the use of VLMs in social robot navigation, no existing work systematically evaluates their ability to meet these necessary conditions. In this paper, we introduce the Social Navigation Scene Understanding Benchmark (SocialNav-SUB), a Visual Question Answering (VQA) dataset and benchmark designed to evaluate VLMs for scene understanding in real-world social robot navigation scenarios. SocialNav-SUB provides a unified framework for evaluating VLMs against human and rule-based baselines across VQA tasks requiring spatial, spatiotemporal, and social reasoning in social robot navigation. Through experiments with state-of-the-art VLMs, we find that while the best-performing VLM achieves an encouraging probability of agreeing with human answers, it still underperforms simpler rule-based approach and human consensus baselines, indicating critical gaps in social scene understanding of current VLMs. Our benchmark sets the stage for further research on foundation models for social robot navigation, offering a framework to explore how VLMs can be tailored to meet real-world social robot navigation needs. An overview of this paper along with the code and data can be found at https://larg.github.io/stars.

1 Introduction

Social navigation requires joint spatial and social reasoning, but existing approaches tend to address one or the other, while grounded HRI data is scarce and fragmented. STARS learns reusable relational representations from interaction scenarios, which support data-efficient downstream tasks and encode social semantics.

1 Introduction

  • Joint spatial and social reasoning is central to HRI, yet existing approaches tend to address only one: spatial models lack social-dynamics understanding, while language and vision-language models struggle with continuous spatial dynamics and precise timing.Training large models from scratch is challenging because grounded HRI data is scarce, fragmented, and prone to overfitting in large unstructured models.Source: §1 · pp. 1–2
  • The paper hypothesizes that graphs suit small, fragmented HRI datasets because nodes represent agents and edges represent interactions, giving message-passing networks a relational inductive bias.This bias is proposed as a way to learn interaction patterns efficiently in limited-data regimes.Source: §1 · p. 2
  • STARS learns event-level relational representations by encoding fixed-duration interaction scenarios as graphs with directed edges for pairwise spatial features over time.The method uses self-supervised learning rather than autoregressive world-model prediction.Source: §1 · p. 2
  • Its graph latents compress physical dynamics while capturing implicit social situations, allowing linear probes to extract social meaning without task-specific representation learning.The demonstrated approach supports data-efficient generalization.Source: §1 · p. 2
  • STARS representations support robust downstream generalization with a fraction of labeled data and transfer across the two evaluated datasets.The paper also reports that latent-space topology distinguishes complex social behaviors, including action intents, without supervised action classification.Source: §1 · p. 2

2 Related Work

Prior social-navigation work models trajectories and interactions, but often prioritizes spatial forecasting over reusable social semantics. Relational graph learning and self-supervised graph representation learning offer relevant tools, while grounded HRI data scarcity remains a challenge for foundation-scale approaches.

Spatial and social reasoning in social navigation

  • Social-navigation studies use pedestrian trajectories and egocentric robot observations, with graph neural networks modeling crowd topology and heterogeneous spatiotemporal GNNs weighting relational dependencies.The passage cites trajectory datasets, egocentric observations [15] [16], GNNs [17] [18], and heterogeneous models [20].Source: §2 · p. 2
  • Trajectory-focused models prioritize future spatial configurations and displacement error, while social tasks require normative behaviors and interpersonal associations that those models struggle to generalize to.The related work cites trajectory-focused models [22] and benchmarks for robot-performance perceptions and VLM understanding of navigation interactions [21] [6].Source: §2 · p. 3

Foundation models and the HRI data bottleneck

  • Foundation and world models show embodied prediction and transfer capabilities, but their reliance on massive unified datasets contrasts with scarce, geographically constrained, fragmented grounded HRI data.The related work reports that data scarcity hinders foundation-scale learning and can lead unstructured models to overfit or fail to generalize across social scenarios.Source: §2 · p. 3

Relational inductive bias

  • Relational inductive bias structures entities as nodes and interactions as edges, and generative self-supervised graph learning can extract meaningful embeddings by reconstructing masked features or topology.The passage notes that applying relational bias to social context remains underexplored [39].Source: §2 · p. 3

3 Method

STARS maps interaction scenarios from diverse social-navigation datasets into common spatiotemporal graphs, learns node-level latents through unsupervised graph autoencoding, and evaluates them with lightweight task-specific probes. Its formulation accommodates variation across datasets and uses relational structure to learn reusable social context.

3.1 Problem Formulation

  • Each interaction scenario is a fixed-duration observation of multiple agents sharing space, represented by agent nodes and temporal pairwise edge-feature sequences within a graph.The graph supports variable numbers of agents and includes a valid mask for observed nodes and edges.Source: §3.1 · p. 3
  • The method targets compact, reusable representations that improve held-out downstream performance under the same labeled-data budget and adaptation protocol.The task labels include subjective human perceptions of robot behavior and discrete pedestrian action categories.Source: §3.1 · p. 4

3.2 STARS Overview

  • STARS constructs unified spatiotemporal graphs, compresses physical dynamics with a self-supervised message-passing autoencoder, then applies lightweight task-specific linear heads to frozen latents.Both homogeneous and heterogeneous formulations are used to study the effect of explicit entity and relation types.Source: §3.2 · p. 4

3.3 Spatiotemporal Relational Graph Construction

  • Graphs emphasize temporal relations in edge features and connect all observed agents rather than applying distance thresholds; homogeneous and heterogeneous variants differ in whether entity and relation types are explicit.The homogeneous variant uses generic agent and interaction types, while the heterogeneous variant types nodes and edges by entity and relation.Source: §3.3 · pp. 4–5

3.4 Unsupervised Graph Autoencoder

  • The MPGNN encodes each node as a latent distribution, and per-node latent representations preserve pairwise asymmetry because edge reconstruction conditions on both endpoint latents.The heterogeneous formulation uses typed-edge message passing and attention to learn which interactions matter; the homogeneous formulation omits edge-type embeddings.Source: §3.4 · p. 5
  • A self-supervised graph autoencoder reconstructs temporal edges from endpoint latents; human nodes can acquire social context through this objective even when they lack node-reconstruction targets.Node-level reconstruction is restricted to nodes with active, non-placeholder input attributes, which in the evaluated datasets means the SEAN-T robot node alone.Source: §3.4 · p. 6

3.5 Task-Agnostic Linear Probing

  • Frozen node latents are selected and aggregated into fixed-dimensional, permutation-invariant task vectors, then a supervised linear probe is trained for the downstream label.The probes use graph-level representations for SEAN-T perception tasks and node-level representations for the SNS action task.Source: §3.5 · p. 6

4 Experiments

The experiments evaluate STARS on downstream social-navigation tasks and test whether its frozen representations are data-efficient and encode action semantics. STARS variants perform competitively with unstructured baselines, and latent-space projections show action-related structure.

4.1 RQ1 & RQ2: Downstream Performance and Data Efficiency

  • The evaluation freezes the pretrained encoder and trains linear probes with progressively larger label fractions, comparing STARS variants with unstructured baselines on SEAN-T and SNS.The reported measures are F1-Scores for SEAN-T competence, surprise, and intention, and macro F1-Score for SNS avoid, follow, and not consider.Source: §4.1 · p. 7, §4 · p. 7
  • STARS-Ho yields the highest overall scores in most SEAN-T categories, while STARS-He achieves the highest SNS macro F1 across labeled-data fractions (0.362 at 1%; 0.503 at 100%).On SNS, unstructured baselines plateau near macro F1-Score 0.3, and STARS-He scores 0.503 versus 0.445 for the supervised model at 100% labeled data.Source: Table 1 · p. 8, §4.1 · p. 8

4.2 RQ3: Latent Space Semantics

  • PCA projections show smooth latent-space variation, while t-SNE reveals visually distinct groupings associated with avoid and follow action labels.These projections provide qualitative evidence of action-relevant structure despite the encoder not being trained with a supervised action-classification objective.Source: Figure 2 · p. 7, §4.2 · pp. 8–9

5 Limitations and Future Work

STARS has task and evaluation boundaries: its learned latents are less suited to edge-centric tasks requiring exact relational geometry, and its evaluation uses fixed-duration windows, processed tracks, and curated datasets.

5 Limitations and Future Work

  • Autoencoding and message passing smooth high-frequency pairwise geometries, making the latents less effective for edge-centric tasks such as group detection that require exact relational geometry.The authors suggest intermediate, pre-message-passing temporal edge embeddings as a way to preserve that geometry.Source: §5 · p. 9
  • The evaluation assumes fixed-duration interaction windows and processed agent tracks, and is limited to curated social-navigation datasets.The authors identify online segmentation, noisier perception, and more diverse real-world settings as future evaluation directions.Source: §5 · p. 9
  • Standardizing diverse datasets into unified graphs can discard dataset-specific nuances.The authors propose more flexible graph construction pipelines as a future direction.Source: §5 · p. 9

6 Conclusion

The conclusion presents STARS as a self-supervised approach that bridges physical dynamics and social reasoning through spatiotemporal graphs and a graph autoencoder. Its evaluations support reusable, data-efficient representations that encode social semantics without direct downstream-task optimization.

6 Conclusion

  • STARS models social-navigation scenarios as spatiotemporal graphs and uses a self-supervised graph autoencoder to bridge physical dynamics and high-level social reasoning.Source: §6 · p. 9
  • The evaluations indicate that STARS representations are reusable and data-efficient, and encode complex social semantics without direct optimization for downstream tasks.Source: §6 · p. 9
  • The authors conclude that structural priors may help offset the scarcity of grounded HRI data.Source: §6 · p. 9

A.1 Datasets

SEAN-T and SocialNav-SUB provide distinct social-navigation data and tasks, which the paper converts into a shared graph-based representation. The appendix describes shared interaction features alongside separate homogeneous and heterogeneous graph configurations.

A.1 Datasets

  • SEAN-T supplies virtual-reality navigation observations paired with subjective human ratings, while SNS supplies robot-centric interaction scenarios with relational action labels.SEAN-T evaluates social impressions; SNS evaluates action classification, including avoid, follow, and yielding.Source: §A.1 · p. 14

B.1.1 Shared Graph Representation

  • Both graph variants represent each fixed-duration scenario with agent nodes and directed edges for ordered pairwise interactions.Each edge stores a temporal sequence of relative spatial and motion features.Source: §B.1.1 · p. 15

B.1.2 Edge Feature Channels

  • Each edge has eight channels: relative pose and heading features plus their per-timestep temporal differences.Differences are zeroed at the first timestep of each contiguous track segment; heading uses sine and cosine, avoiding wrap-around discontinuities.Source: §B.1.2 · p. 15

B.1.3 Homogeneous Graph

  • The homogeneous graph uses a common 40-timestep window and 592-dimensional node feature space, with robot and human identity represented by a two-class node embedding.SNS is interpolated from 20 to 40 steps; a temporal edge transformer and homogeneous TransformerConv process the graph.Source: §B.1.3 · p. 15

B.1.4 Heterogeneous Graph

  • The heterogeneous graph separates robot and bystander nodes and uses three directed edge types, retaining native temporal resolutions for SEAN and SNS.Its node features combine learnable embeddings with positional features, and both graph configurations use eight-dimensional edge features.Source: §B.1.4 · pp. 15–16, Table 3 · p. 16

B.2.1 Per-Dataset and Shared Components

STARS handles differences between datasets with variant-specific sharing, then encodes temporal relations and agent features for relational message passing. The heterogeneous and homogeneous variants use different message-passing designs.

B.2.1 Per-Dataset and Shared Components

  • The heterogeneous variant keeps input and output adapters per dataset while sharing the relational message-passing core and latent heads; the homogeneous variant shares every component after dataset conversion.The heterogeneous design uses four node feature projections, three learned type embeddings, two temporal edge encoders, and two edge decoders.Source: §B.2.1 · p. 16, Table 4 · p. 16

B.2.2 Feature Preprocessing

  • Node attributes are projected by type-specific MLPs, while a Transformer with learned positional embeddings compresses each temporal edge sequence into a fixed-dimensional relation embedding.These preprocessing steps place node and edge features in a shared hidden dimension.Source: §B.2.2 · pp. 16–17

Homogeneous Message Passing

  • The homogeneous encoder instead uses shared edge-conditioned attention convolution, placing encoded temporal edge features in the key and value terms without an edge-type embedding.Source: Homogeneous Message Passing · p. 17

Heterogeneous Message Passing

  • Heterogeneous message passing uses a HEAT-style update, parameterizing attention and messages with an edge-conditioned interaction representation and updating node states with residual connections and Layer Normalization.Edge embeddings also receive residual updates using message content already computed for node aggregation.Source: Heterogeneous Message Passing · p. 17, Figure 3 · p. 18

C Training and Evaluation Protocol

The protocol pretrains graph encoders with a self-supervised objective, freezes them for task-specific linear probes, and averages results across random seeds. Unified-model tuning and adaptive sampling address downstream task and class-imbalance considerations.

C Training and Evaluation Protocol

  • Experiments use standard feature scaling, early stopping with patience 20, AdamW, and results averaged across 10 random seeds.Implementations use mixed precision; model-specific learning rates and selected hyperparameters are reported in Appendix C.3.Source: C Training and Evaluation Protocol · p. 19

C.1 Pretraining Protocol

  • Models are pretrained with the ELBO self-supervised VAE objective, processing each interaction graph independently so scenarios can contain variable numbers of human nodes and edges.Source: §C.1 · p. 19

C.2 Downstream Adaptation Protocol

  • After pretraining, the encoder is frozen and each downstream task is trained with a lightweight linear prediction head on extracted latent graph representations.SEAN-T probes use robot and mean-human latents; SNS probes pair the robot latent with the labeled pedestrian's latent, and report macro-F1 over avoid, follow, and not consider.Source: §C.2 · p. 19

C.3 Hyperparameter Search

  • Optuna tuning was limited to unified SEAN+SNS models, with SNS probe validation loss as the objective; heterogeneous searches varied βmax, and the best validation performance came from βmax = 0.9.The SEAN-only baseline reused selected unified hyperparameters rather than receiving a separate search.Source: §C.3 · pp. 19–20

C.4 Sampling and Class Imbalance

  • Unified training used stratified and adaptive sampling to address SNS imbalance, where approximately 82% of training examples are not consider; this sampler uses downstream SNS label statistics despite the unsupervised VAE objective.The strategy improved downstream F1 scores on both SEAN and SNS, with particularly noticeable gains for the homogeneous model.Source: §C.4 · p. 20

D Compute and Reproducibility Details

The extended results show performance generally rising with more labeled data, with a heterogeneous-model advantage on SNS macro-F1 at larger fractions. The appendix also documents computing conditions, model capacity, runtime measures, and SEAN-only pretraining results.

D Compute and Reproducibility Details

  • The reproducibility details cover dataset sizes, graph construction, model capacity, training and inference runtimes, and hardware; experiments primarily ran on Lonestar6 and Stampede3, usually on a single 40 GB NVIDIA A100 PCIe GPU.The stated software environment was Ubuntu 22.04, Python 3.10, PyTorch 2.0.1, and CUDA-enabled GPU execution.Source: D Compute and Reproducibility Details · p. 21, §D.1 · p. 21

D.2 Model Capacity

  • STARS capacity is reported separately for the pretrained representation model and the downstream task-specific prediction head.Source: Table 5 · p. 21, §D.2 · p. 21

D.3 Runtime and Memory

  • Runtime reporting distinguishes graph-autoencoder pretraining, task-head fitting on the frozen encoder, and inference per interaction scenario, including representation extraction and prediction when applicable.Peak GPU memory is measured during training unless otherwise stated.Source: §D.3 · p. 21, Table 6 · p. 21

E.1 Extended Main Results

  • Across SEAN-T and SNS, performance generally improves with more labeled data; the variants are similar on SEAN-T, while the heterogeneous model achieves higher SNS macro-F1 at larger label fractions.Source: Table 7 · p. 22, §E.1 · p. 22

E.2 Pretraining Data Ablations

  • At 100% labeled data, SEAN-only pretraining scores 0.792/0.716/0.721 on competence, surprise, and intention, versus 0.794/0.709/0.721 under joint pretraining, indicating no substantial SEAN-T performance change from adding SNS data.Source: Table 8 · p. 22, §E.2 · p. 22

E.3 Cross-Dataset Transfer

Cross-dataset transfer is comparable to joint pretraining for STARS-He at full labels, while ablations favor pretrained reconstruction-based representations over random or supervised alternatives.

E.3 Cross-Dataset Transfer

  • At 100% labels, SEAN-T-only scores 0.497 macro F1 on SNS (joint: 0.503); SNS-only scores 0.801/0.710/0.729 on SEAN-T competence/surprise/intention (joint: 0.794/0.709/0.721).STARS-Ho also transfers across datasets, although joint pretraining provides larger gains.Source: §E.3 · p. 22, Table 9 · p. 23, Table 10 · p. 23

E.4 Pretraining Objective Ablations

  • The random encoder shares the full heterogeneous GNN-VAE architecture but performs worse on most F1-scores, indicating that self-supervised reconstruction learns task-relevant latent structure.Source: §E.4 · p. 23

E.4.1 Self-Supervised vs. Supervised Heterogeneous Graph

  • Across label fractions, self-supervised pretraining achieves stronger macro-F1, although supervised training improves some minority-class scores such as avoid.The comparison suggests reconstruction-based pretraining provides a more stable representation for low-data adaptation under SNS class imbalance.Source: §E.4.1 · p. 23

E.5 Representation Component Ablations

Representation ablations show complementary value in robot and bystander features, while the lower-bound analysis highlights class-imbalance pitfalls; VLM baselines are assessed on the held-out SNS action task.

E.5 Representation Component Ablations

  • Bystander-only representations retain task-relevant signal, but combining robot and pedestrian representations performs better overall.Source: §E.5 · pp. 23–24
  • At 100% labels, raw robot-node scores 0.746/0.700/0.688 on competence/surprise/intention versus full STARS’ 0.794/0.709/0.721; robot-latent-only trails full on minority SNS actions and most SEAN-T perception tasks.Raw features outperform robot-latent-only on competence and surprise but underperform on intention.Source: §E.5 · pp. 23–24, Table 13 · p. 25, Table 15 · p. 26

E.6 Baseline and Lower-Bound Analyses

  • The weighted-random baseline can score strongly on majority labels such as not consider, making accuracy or single-class F1 misleading under SNS class imbalance.The full model substantially outperforms this baseline in nearly every F1 metric.Source: §E.6 · p. 24

E.7 SNS F1-Score Analysis

  • Zero-shot VLM baselines predict discrete relational actions for target pedestrians and are evaluated on the full held-out SNS test set using per-class F1 and Probability of Agreement.Source: §E.7 · pp. 24–25, Table 17 · p. 26

Related papers

Loading 2609.40245v2…