PaperlayerView exact revision on arXiv

Source-linked AI summary

Social Intuition vs. Machine Reasoning: Anticipating Human-Robot Interaction from multiple modalities

Raphael Lorenzo-Louis, Bertrand Luvison, Serena Ivaldi

arXiv:2609.07394v1cs.CV

TL;DR

The paper studies how humans and machine models anticipate whether someone will interact with a service robot, comparing pose and egocentric video on HUI360. Humans outperform pose models with pose input and gain substantially with video, while VLM performance does not consistently track model size.

  • Problem

    Social robots need reliable early predictions of whether someone will interact so they can plan motion and address people willing to engage.

    Source: §I · p. 1

  • Method

    The study benchmarks human annotators, trained pose models, and VLMs on HUI360 using pose and video input conditions.

    Source: §I · p. 1, A. Dataset · p. 2, D. Vision-language models · p. 3

  • Results

    Humans outperform pose models on Pose and reach mean F1 0.77 ± 0.08 on Vbbox, while VLMs often underuse social-temporal cues and performance shows no clear correlation with model size.

    Source: §IV · p. 3, §V · p. 4

  • Takeaways & Limitations

    Trained pose models such as ST-GCN remain an acceptable lightweight alternative when compute is limited.

    Source: §V · p. 4

    The results are preliminary; future work should compare VLMs with HUI360-trained visual models and further investigate model failures and human judgment.

    Source: §V · p. 4

Abstract

from arXiv · show

Anticipating whether a person will interact from one's own perspective is a highly intuitive task for humans, that relies on a combination of cues. We investigate how humans perform at predicting a person's intention to interact from a service robot's point of view, using pose-only or full video input, then benchmark different lightweight pose-based models and state-of-the-art vision-language models. We conducted our benchmark on the HUI360 dataset on a fixed pilot subset of 100 test tracks (25 positive, 75 negative). We found that with pose-only input, human annotators outperform lightweight trained pose models but not by large margins (+0.08 in F1-Score). But when given full egocentric video with a target bounding box, human annotators perform substantially better and largely outperform the Vision-Language Models (+0.2 in F1-Score). We also compared VLMs of different size and under different input conditions, and found that the best results do not correlate with model size. Our result confirms that predicting interactions is a challenging task for social robots and that reasoning-capable models are necessary but their actual reasoning capabilities alone do not suffice to match the social intuition of humans.

I. INTRODUCTION

Social robots need to anticipate who will interact so they can plan behavior and address people willing to engage. The study compares human and machine predictions across representations, modalities, and model types.

I. INTRODUCTION

  • Reliable interaction prediction matters for robot motion planning and for addressing only people willing to interact.The paper frames this as a binary prediction of whether a person will interact.Source: §I · p. 1
  • Vision-language models are computationally expensive and high-latency, whereas lightweight 2D pose can run in real time on edge devices.Source: §I · p. 1
  • The study examines how interactant representation affects anticipation performance for humans and vision models across modalities and machine-learning approaches.Source: §I · p. 1
  • The fixed pilot benchmark uses 100 HUI360 test tracks—25 positive and 75 negative—with a one-second pre-interaction cutoff and Pose and Vbbox conditions for human annotators.Pose overlays 2D pose on static scene context; Vbbox shows egocentric video with a target bounding box.Source: §I · p. 1
  • Pose-trained sequence baselines include LSTM, ST-GCN, and SkateFormer; VLM comparisons vary model, input condition, thinking mode, sampling rate, prompt length, and projection style.The VLM input conditions are image crop, video with image reference, and video with a per-frame target box.Source: §I · p. 1

II. RELATED WORK

Prior work covers egocentric interaction datasets and human anticipation benchmarks, alongside VLM applications in social navigation and perception. Reported results also identify challenges in video-based emotion and social-cue reasoning.

II. RELATED WORK

  • HUI360 is described as the largest in-the-wild egocentric human–robot interaction anticipation dataset, while earlier benchmarks are smaller or less reproducible.PAR-D aggregates data from two robots in public environments; prior human F1-scores ranged from 0.65 to 0.86 and varied by observation length and deployment context.Source: §II · p. 1
  • Some HRI studies use video VLMs as commonsense proxies for deciding when to engage, often after lightweight preamble detectors [6].Source: §II · p. 1
  • VLM-Social-Nav [7] uses a vision-language model to score candidate robot actions for socially compliant trajectories without task-specific navigation-dataset training.Source: §II · p. 2
  • Prior VLM studies report difficulty with video emotion recognition, including biases and limited high-frequency temporal modeling [8], [9].A separate comparison found strong context-based anticipatory judgments but weaker use of social cues.Source: §II · p. 2

A. Dataset

The study uses HUI360, an egocentric 360° dataset from a static tray-holding robot, and evaluates a fixed set of test tracks with physically grounded interaction labels.

A. Dataset

  • HUI360 uses Shelfy, a static tray-holding robot, across nine indoor environments; people are tracked with SAM2, and tray interactions are automatically annotated then manually reviewed.The HUI split contains 4,310 samples, including 375 interactions; the complementary SSUP-A split is excluded.Source: A. Dataset · p. 2
  • The 100 HUI-Test tracks include 25 positive tray pick-or-place actions and 75 negatives, with two environments unseen by the pose models.Positive windows end about one second before interaction onset; negative windows end at closest approach, making them harder than simple walk-away cases. Gaze, hesitation, and speech are cues, not labels.Source: A. Dataset · p. 2

B. Human baseline

Human annotators judged the same 100 tracks using either pose over static scene context or egocentric video with a target bounding box. The task asked whether the highlighted person would interact with the robot’s tray within two seconds.

B. Human baseline

  • Six annotators judged tracks under Pose and Vbbox conditions, but not all completed both; each completed condition covered all 100 tracks, including difficult trials.Three annotators labeled both conditions, two only Vbbox, and one only Pose, yielding n=4 and n=5 for the respective conditions.Source: B. Human baseline · p. 2
  • For each randomized clip, annotators predicted whether the highlighted person would take or place something on the tray within two seconds.Pose includes static RGB context to ground the skeleton in the equirectangular scene.Source: B. Human baseline · p. 2

C. Baseline skeleton models

The section describes three lightweight skeleton-based baselines evaluated on HUI360 pose input. Their benchmark reports classification metrics and per-sequence latency.

C. Baseline skeleton models

  • Table I reports F1-score, accuracy, precision, recall, and per-sequence latency for models trained on HUI360-Train and evaluated on the 100-track subset.Latency is measured on an RTX 2000 Ada GPU for a full 32-frame sequence, excluding 2D-pose extraction.Source: C. Baseline skeleton models · p. 2
  • The LSTM, ST-GCN, and SkateFormer baselines use 32-frame sequences of 17 COCO joints, with parameter counts of 0.37M, 3.07M, and 1.91M, respectively.ST-GCN uses spatial-temporal graph convolution with learnable edge-importance weighting; SkateFormer applies self-attention over explicit skeleton partitions.Source: C. Baseline skeleton models · p. 2
  • Training follows the HUI360 protocol with a 16-frame anticipation cutoff and weighted binary crossentropy; the models use AdamW with a learning rate of 10−3.LSTM and ST-GCN train for 100 epochs, while SkateFormer trains for 600 epochs with gradient clipping.Source: C. Baseline skeleton models · p. 2

D. Vision-language models

The section compares six vision-language models across image, video-reference, and bounding-box input conditions. It also describes their prompting, sampling, and per-track latency setup.

D. Vision-language models

  • The benchmark compares six VLMs, from Qwen3.5-0.8B and InternVL3-1B to larger Qwen and Gemini models, under Ic, V + Ic, and Vbbox conditions.Fig. 2 depicts the VLM prompting setup and an ablation illustration.Source: D. Vision-language models · p. 2, Figure 2 · p. 3
  • Ic uses a cropped target image from 16 frames before onset, while V + Ic pairs full video with a target reference image and Vbbox supplies target-box coordinates with the video.The models tested tracked the correct target across frames in V + Ic; Vbbox uses normalized coordinates from 0 to 999.Source: D. Vision-language models · pp. 2–3
  • Unless noted otherwise, VLMs receive video sampled at 8 FPS from 960×480 equirectangular videos, while image crops come from 3840×1920 frames.Runs enable thinking mode with a 1024-token budget; latency is reported as mean API or on-device processing time per track.Source: D. Vision-language models · p. 3

IV. PRELIMINARY RESULTS

Humans outperform the tested models on interaction anticipation, with a much larger advantage when given video than pose alone. VLM performance varies with input condition, while the ablations identify several influential settings.

Visual modalities

  • Video does not consistently help larger VLMs: Gemini 3.6-Flash peaks on V + Ic (F1 0.53), while Qwen3.5-35B-A3B and Qwen3.8-Max perform best with Ic.Smaller on-device models score F1 0.23–0.35 on Ic and F1 0.40-0.44 under Vbbox.Source: §IV · p. 3
  • Humans reach mean F1 0.77 ± 0.08 (best: 0.90) with Vbbox, versus the strongest zero-shot VLM result, Qwen3.5-35B-A3B at F1 0.57 on Ic.VLMs given video often underperform, with some producing high-recall, low-precision predictions.Source: Table I · p. 3, §IV · p. 3

Pose modality

  • For Pose, human annotators reach mean F1 0.57±0.02 (best: 0.59), ahead of the top pose model, ST-GCN, at F1-score 0.49.The passage attributes the human advantage to inferring subtle cues from coarse body dynamics.Source: Table I · p. 3, §IV · p. 3

Prediction correlations

  • Human prediction agreement is higher under Pose than visual input (average Pearson r 0.42 versus 0.26), while VLM correlations vary across model families and modalities.For large models, image-input correlations range r 0.38−0.62 within families and r 0.32 −0.78 across families.Source: §IV · pp. 3–4, Figure 3 · p. 4

Ablations

  • Ablations show thinking mode is necessary, 2-to-8-FPS sampling improves F1, equirectangular clips outperform pinhole reprojection, and longer prompts have only modest effect.The authors suggest that preserving the whole visual context, including the interaction zone, is necessary; the short prompt is slightly preferred.Source: §IV · p. 4
  • For Qwen3.8-Max under Vbbox, increasing the reasoning-token budget from 1024 to 8192 raises F1 from 0.48 to 0.51 and recall from 0.84 to 0.96.Source: §IV · p. 4

V. CONCLUSION AND DISCUSSION

Humans outperform pose models using Pose and gain substantially from full video, while VLMs struggle with social dynamics and performance shows no clear correlation with model size. The authors characterize these findings as preliminary and identify further comparisons and user studies.

V. CONCLUSION AND DISCUSSION

  • Humans outperform pose models on Pose and gain substantially from full video, while VLMs struggle with social dynamics despite thinking mode.Source: §V · p. 4
  • No clear correlation between model size and performance is observed; ST-GCN remains an acceptable lightweight alternative when compute is limited.Source: §V · p. 4
  • Table II covers Vbbox ablations on 100 tracks; Qwen3.8-Max latency and token columns report mean ± standard deviation.Source: Table II · p. 4
  • Qualitative examples distinguish motion-related thinking, visual-social reasoning, certainty, dataset references, and reasoning about hands or arms in VLM answers.Source: Figure 4 · p. 4
  • The authors call the results preliminary and propose HUI360-trained lightweight image/video comparisons, failure-locus probing, and a deeper human study.Source: §V · p. 4
Loading 2609.07394v1…