PaperlayerView exact revision on arXiv

Source-linked AI summary

SocioGesture: Real-Time and Adaptive Social Gesture Perception for Human-Robot Interaction

Wenjin Fu, Li-Fan Wu, Jerin Peter, Chip Huyen, Boyuan Chen, Jan Liphardt

arXiv:2609.04545v1cs.ROcs.CV

TL;DR

Robots need to recognize social cues from noisy, partly occluded onboard perception without delaying interaction. SocioGesture combines lightweight body-hand skeleton recognition with occlusion-aware training and offline adaptation of uncertain segments. It reports improved structured-occlusion accuracy alongside held-out-subject recognition, real-time edge deployment, and vocabulary expansion while preserving original-class performance.

  • Problem

    Social gestures can signal invitations, refusals, or unavailability, but robot-view recognition must contend with occlusion, changing viewpoints, unstable keypoints, and strict latency constraints.

    Source: §1 · pp. 1–2

  • Method

    SocioGesture uses confidence-aware body-hand skeleton encoding and a lightweight single-pass model, trains with occlusion-aware corruption, and sends uncertain interaction segments for offline labeling and adaptation.

    Source: §1 · p. 3, §3.2 · p. 5

  • Results

    Hand-occlusion accuracy improves from 31.8% to 84.9%, and arm-occlusion accuracy from 28.3% to 79.5%; experiments also report held-out-subject recognition, real-time edge deployment, and vocabulary expansion with original-class performance preserved.

    Source: §4.2 · p. 7, §6 · p. 11

  • Takeaways & Limitations

    The results suggest a practical path toward social perception systems that are lightweight, robust to noisy deployment, and adaptive after deployment.

    Source: §6 · p. 11

    The current recognition window may not support longer activities, multi-step exchanges, or context-dependent gestures, and severe occlusion can remove the hand or arm evidence needed to distinguish gestures.

    Source: §5 · p. 10

Abstract

from arXiv · show

Robots interacting with people must recognize not only explicit commands, but also social cues such as invitations, refusals, and unavailability. In real deployments, these cues must be inferred from noisy onboard perception under partial occlusion, changing viewpoints, and strict latency constraints. We present SocioGesture, a real-time adaptive social gesture perception system for human-robot interaction (HRI). SocioGesture uses a compact confidence-aware body-hand skeleton representation and a lightweight dual-stream model that fuses body motion with hand articulation for low-latency onboard recognition. To improve deployment robustness, we train the model with occlusion-aware skeleton corruption, exposing it to missing hands, occluded arms, and temporally unstable keypoints without increasing the inference cost. On a social gesture dataset collected in mixed indoor-outdoor HRI scenarios, SocioGesture achieves strong held-out-subject recognition, substantially improves robustness under structured joint occlusion, and runs in real time on a robot-mounted edge device. During deployment, uncertain interaction segments are saved for offline labeling and adaptation, enabling SocioGesture to expand its gesture vocabulary while preserving performance in the original classes. These results demonstrate a practical path toward robust, efficient, and adaptive social perception for interactive robots.

1 Introduction

Social gestures can shape robot behavior before a verbal command, so recognition must be safe and reliable under noisy, changing onboard perception. SocioGesture keeps recognition lightweight and local while routing uncertain segments to offline labeling and adaptation.

1 Introduction

  • Unlike offline action classification, HRI recognition directly affects robot behavior, and SocioGesture defers uncertain decisions rather than forcing potentially unsafe actions.A false invitation can trigger an unwanted approach, while a missed stop signal may fail to halt the robot.Source: §1 · p. 2
  • Robot-view gesture recognition must handle changing viewpoints, self-occlusion, keypoint failures, and latency constraints while remaining able to improve from open-ended deployment interactions.Hands and arms carry many social cues but are small and often occluded; large video models may be too slow or cloud-dependent, while small onboard models may be brittle.Source: §1 · p. 2
  • SocioGesture combines confidence-aware body-hand pose, a lightweight single-pass fusion model, occlusion-aware training, conservative robot behavior, and uncertainty-driven offline adaptation.The system also uses stronger video understanding models offline to label uncertain segments for adaptation.Source: Figure 1 · p. 2, §1 · p. 3
  • The authors plan to release the dataset and evaluation code to support research on deployable HRI perception.Source: §1 · p. 3

2 Related Work

Prior work supplies skeleton-based recognition, robustness techniques, and deployment-adaptation ideas, but SocioGesture focuses on combining them for robot-view social gestures in a closed-loop setting. Its approach uses compact body-hand recognition, training-time corruption, and offline labeling of uncertain interactions.

2.1 Skeleton-Based Gesture and Action Recognition

  • Skeleton-recognition methods model joint sequences with graph networks, improved training and representations, heatmaps, or body-hand articulation, but are generally evaluated on curated benchmarks rather than robot behavior and deployment failures.SocioGesture instead uses a compact confidence-aware body-hand model in one forward pass for low-latency edge inference.Source: §2.1 · p. 3

2.2 Robust and Deployable Gesture Perception for HRI

  • Prior HRI gesture systems map body cues to robot motion and study timely partial-observation recognition, while deployment pose estimates can be incomplete or unstable.SocioGesture uses occlusion-aware skeleton corruption to improve robustness to missing hands, occluded arms, and unstable keypoints without added inference computation.Source: §2.2 · pp. 3–4

2.3 Foundation Models and Deployment Adaptation

  • Foundation models offer semantic interpretation but can add latency, compute cost, and dependence in time-critical HRI; active and continual learning offer routes to postdeployment improvement.SocioGesture runs a lightweight skeleton model onboard and uses a stronger video model offline to label uncertain segments for fine-tuning.Source: §2.3 · p. 4

3 Method

SocioGesture separates real-time interaction from offline adaptation while using a shared skeleton representation across both paths.

3 Method

  • The online path predicts gestures from tracked-person skeleton windows, maps confident predictions to conservative behaviors, and saves uncertain segments for later labeling and fine-tuning.The offline path uses the same skeleton representation as the online path.Source: §3 · p. 4, Figure 2 · p. 5

3.1 Confidence-Aware Skeleton Encoding

  • Each joint is encoded with its coordinate, confidence, motion, bone vector, and bone motion, retaining partially observed windows instead of discarding unreliable joints.Coordinates are normalized within each tracked-person window; confidence distinguishes an unreliable or missing hand from a stationary visible hand.Source: §3.1 · p. 4

3.2 Body-Hand Gesture Model and Uncertainty

  • Separate lightweight body and hand streams exchange temporally aligned features through cross-attention, then a fused head classifies them in one end-to-end forward pass.Auxiliary body and hand heads provide training supervision and per-stream predictions for the uncertainty score.Source: §3.2 · pp. 4–5
  • The deployment uncertainty score selects informative segments using fused-prediction entropy, body–hand disagreement, and average skeleton confidence.The selection score is separate from the confidence threshold used to choose robot behaviors.Source: §3.2 · pp. 5–6

3.3 Occlusion-Aware Training and Offline Adaptation

  • Training corrupts skeleton windows with structured hand and arm masking, temporal hand disappearance, and random joint masking, then recomputes motion-related channels.Occlusion-aware training adds no inference-time computation.Source: §3.3 · p. 6
  • For offline adaptation, uncertain deployment segments are labeled and filtered, new-class weights are initialized, and fine-tuning mixes pseudo-labeled deployment windows with original data for rehearsal.The updated model is redeployed only after held-out validation.Source: §3.3 · p. 6

4 Experimental Evaluation

The evaluation spans held-out-subject recognition, structured occlusion, edge latency, offline vocabulary adaptation, and live closed-loop trials. Results show improved occlusion robustness and high live recognition, alongside more conservative behavior success.

4.1 Setup

  • The HRI dataset contains seven gestures from invitation, prohibition, unavailability, and non-interaction categories, with 7,055 skeleton windows from 10 subjects.Evaluation uses leave-one-subject-out cross-validation; adaptation is assessed on a separate held-out five-subject test set with a 10-class vocabulary.Source: §4.1 · p. 7

4.2 Recognition and Occlusion Robustness

  • Hand-occlusion accuracy improved from 31.8% to 84.9%, and arm-occlusion accuracy improved from 28.3% to 79.5%.The standard model achieved 96.6%±2.8% clean accuracy; occlusion-aware variants stayed within one percentage point with overlapping fold-level variability.Source: §4.2 · p. 7, Table 1 · p. 8

4.3 Edge Latency and Offline Adaptation

  • Onboard, full-stack deployment latency was 25 ms per frame for one person, without online cloud inference, meeting the 33 ms budget at the 30 FPS camera rate; three-person latency was 42 ms.Source: §4.3 · p. 7, Table 2 · p. 8
  • Offline adaptation raised 10-class clip accuracy from 64.6% to 87.9% and macro F1 from 56.7% to 88.5%, retaining 98.5% accuracy on the original seven classes.Accuracy on the three newly added classes reached 67.6%; adaptation fine-tuned on original human-labeled skeletons mixed with 67 pseudo-labeled deployment clips.Source: §4.3 · p. 8, Table 3 · p. 9

4.4 Efficiency Against Skeleton Recognition Baselines

  • SocioGesture uses 1.3M parameters and 2.3 GFLOPs in a single forward pass, a practical efficiency regime for robot deployment.The comparison contextualizes computational efficiency against skeleton-recognition methods rather than claiming state-of-the-art NTU accuracy.Source: §4.4 · p. 9, Table 4 · p. 10

4.5 Live Closed-Loop Deployment

  • 97.3% of live gestures were recognized correctly (146/150), while behavior success was 79.3%; among acted-on trials, behavior was correct in 98.3%.Adapted gestures transferred at 44/45 trials versus 102/105 for original gestures; no stop gesture caused the robot to approach, and conservative non-action was the dominant failure mode.Source: §4.5 · pp. 9–10, Table 5 · p. 10

5 Limitations and Future Work

SocioGesture keeps online interaction lightweight and conservative, while adapting offline from uncertain segments. Its scope is short social gestures; severe occlusion can remove the evidence needed to distinguish them.

5 Limitations and Future Work

  • Longer activities, multi-step exchanges, and context-dependent gestures may require temporal segmentation and memory beyond the current recognition window.Severe occlusion can remove hand or arm evidence needed to distinguish gestures; low-confidence deferral cannot recover information absent from the input.Source: §5 · p. 10
  • Offline video labeling and human verification reduce incorrect pseudo-label updates but limit how quickly the system can learn from large interaction datasets.The authors identify scalable verification and continual adaptation as future work.Source: §5 · pp. 10–11
  • Gestures are short-horizon visual cues rather than complete social intent; future systems should combine them with speech, gaze, task state, and interaction history.The lightweight onboard path should be preserved as these context sources are added.Source: §5 · p. 11

6 Conclusion

SocioGesture combines confidence-aware body-hand encoding, lightweight temporal fusion, occlusion-aware training, conservative behavior selection, and uncertainty-driven offline adaptation. Experiments report held-out-subject recognition, occlusion robustness, real-time edge deployment, vocabulary expansion, and successful live robot behavior.

6 Conclusion

  • SocioGesture combines confidence-aware body-hand skeleton encoding, lightweight temporal fusion, occlusion-aware training, conservative behavior selection, and uncertainty-driven offline adaptation.Source: §6 · p. 11
  • Experiments show held-out-subject recognition, structured-occlusion robustness, real-time edge deployment, vocabulary expansion from verified clips, and successful live robot behavior.The conclusion describes these results as a practical path toward lightweight, robust, adaptive social perception for HRI.Source: §6 · p. 11

A Additional NTU Occlusion Stress Test

The NTU cross-subject experiments use RTMW-extracted 2D 59-joint skeletons and a simplified single-stream ST-GCN++ setup as a robustness stress test, not as the primary HRI result or an identical-input comparison to SocioGesture. Occlusion-aware training improves corrupted-skeleton robustness while leaving clean accuracy essentially unchanged.

A Additional NTU Occlusion Stress Test

  • This cross-subject NTU RGB+D 60 stress test uses RTMW-extracted 2D skeletons and simplified single-stream ST-GCN++, not the primary HRI dual-branch recognizer.The setup isolates confidence-aware input channels, occlusion-aware training, and CWGCN; it is not an identical-input comparison to the main recognizer.Source: A Additional NTU Occlusion Stress Test · p. 14
  • R20/R40/R60 mask 20%, 40%, and 60% of joints randomly; Hand masks both hands, Arm masks one arm and its connected hand, and Temp-Hand masks hands over a contiguous temporal span.Source: A Additional NTU Occlusion Stress Test · p. 14
  • Occlusion-aware training improves robustness substantially while clean accuracy stays essentially unchanged (94.4% →94.5%).Standard models collapse under arm occlusion and strong random corruption; examples include 9-channel arm accuracy of 27.9% and R60 accuracy of 2.5%.Source: A Additional NTU Occlusion Stress Test · p. 14, Table 6 · p. 14
  • Occlusion consistency further improves the most severe random-corruption cases, while the largest gains come from corrupted-skeleton training rather than the confidence-gated graph architecture.Source: A Additional NTU Occlusion Stress Test · p. 14

B Implementation Details

The implementation table summarizes fixed hyperparameters for training, deployment, and adaptation.

B Implementation Details

Related papers

Loading 2609.04545v1…