PaperlayerView exact revision on arXiv

Source-linked AI summary

SocialVLA: A Social Perception Gateway for Human-Reaction-Based Failure Detection and Recovery in VLA Manipulation

Sofya Konstantinova, Miguel Altamirano Cabrera, Artem Lykov, Dzmitry Tsetserukou

arXiv:2610.02360v1cs.RO

TL;DR

VLA policies can fail during execution without recognizing their own errors, while observers may react to unexpected behavior before a failure is complete. SocialVLA turns those reactions into local runtime hold requests and records verbal corrections; offline results show increased reaction coverage with relevance estimation reducing false stops, alongside measured live stopping latency.

  • Problem

    VLA policies can fail during manipulation without recognizing execution errors, and human reactions offer complementary signals that may occur before failure completion.

    Source: §I · pp. 1–2

  • Method

    SocialVLA is a local, policy-agnostic gateway that fuses causal audio, visual reactions, explicit stop phrases, and robot relevance to request a VLA hold, then logs participant corrections separately.

    Source: §I · p. 2, §VIII · p. 8

  • Results

    Audio-video fusion reaches 64.3% recall, while relevance estimation reduces false stops from 100 to 57 and raises precision from 60.5% to 69.8%.

    Source: C. Offline Ablation Study · p. 6

  • Takeaways & Limitations

    The results demonstrate a local pathway from spontaneous human reactions to physical VLA interruption, with synchronized reaction and correction records potentially useful for future policy learning.

    Source: §VIII · p. 8

    SocialVLA detects human-perceived intervention need rather than all physical failures, and it is not a certified safety system or replacement for robot-state detection.

    Source: §VII · p. 8

Abstract

from arXiv · show

Vision-language-action (VLA) policies enable diverse robotic manipulation but can fail during execution without recognizing their own errors. Human observers provide complementary signals, as unexpected robot behavior can trigger rapid vocal, facial, or verbal reactions before failure is completed. We introduce SocialVLA, a local, policy-agnostic social perception gateway that converts spontaneous human reactions into runtime intervention signals for VLA manipulation. SocialVLA combines causal paralinguistic audio detection, visual reaction recognition, explicit stop phrases, and robot-relevance estimation. An asynchronous first-event fusion mechanism triggers a VLA hold from the earliest sufficiently confident signal, while a separate speech channel captures verbal corrections for participant-directed continuation, restart, or instruction revision. We evaluate SocialVLA on physical Unitree G1 manipulation using 15 participants, with 238 annotated intervention-worthy episodes and 1.038 h of non-intervention behavior. Frozen offline replay achieves 54.6% recall and 69.5% precision, while unfiltered audio-video fusion reaches 64.3% recall. Relevance estimation reduces false-stop episodes from 100 to 57 and increases precision from 60.5% to 69.8%. In prospective deployment on an unseen 16th participant, the frozen system achieves 59.5% recall and 91.7% precision. Median detector-to-fusion latency is 47.9 ms, VLA-gate-to-physical-hold latency is 336 ms, and reaction-onset-to-hold latency is 1.021 s. These results demonstrate a complete local pathway from spontaneous social reaction to physical VLA interruption and participant-directed recovery.

I. INTRODUCTION

VLA execution failures can remain unrecognized, while human reactions offer timely evidence that existing robot-centered monitors may not capture. SocialVLA turns those reactions into local, policy-independent intervention events for a continuously executing VLA.

A. Failure Detection for Robot Policies

  • Robot-centered failure monitors use policy features, sequential out-of-distribution detection, or vision-language judgments, but may depend on internal access, compatible observations, training data, or task-progress models.SocialVLA instead detects human-perceived intervention need and does not require VLA activations, action probabilities, or internal uncertainty estimates.Source: §I · pp. 1–2

I. INTRODUCTION

  • Human observers may react to unexpected robot behavior through stop phrases, vocal prosody, facial expressions, head motion, or redirected attention, sometimes before a failure is complete.These responses reflect the observer’s appraisal and may reveal discrepancies not enumerated during policy training.Source: §I · p. 2
  • SocialVLA addresses the gap between retrospective reaction classification and a runtime gate that must act early, tolerate missing modalities, and avoid unnecessary stops.It is local, policy-agnostic, and designed to request a hold from the earliest reliable signal without waiting for multimodal agreement.Source: §I · p. 2
  • The proposed gateway combines causal audio, visual reactions, explicit stop commands, robot relevance, and asynchronous first-event fusion, then evaluates the system on a physical Unitree G1 with offline and prospective participants.It complements robot-internal monitors by adding the observer’s task-level assessment while remaining independent of the underlying VLA architecture.Source: §I · p. 2, A. Failure Detection for Robot Policies · p. 2

B. Human Reactions to Robot Errors

Prior HRI work links human responses to robot errors, while SocialVLA targets live intervention during VLA execution and records reactions alongside subsequent corrections. Its event representation distinguishes reaction timing, source modality, and diagnostic attention estimates.

B. Human Reactions to Robot Errors

  • Prior work studies multimodal responses and reaction benchmarks, but unrelated stimuli, groups, and repeated failures can introduce noise in less controlled environments.SocialVLA is positioned as an external gateway for a continuously executing VLA, separating reaction recognition from robot relevance and allowing a sufficiently strong unimodal signal to trigger a hold.Source: B. Human Reactions to Robot Errors · pp. 2–3

C. Human Reactions as Temporal Supervision

  • Synchronized stop records associate the active VLA action or chunk with a participant’s verbal correction, creating reaction-centered records that may support future failure-aware or preference-based policy learning.The system records this evidence without requiring the observer to operate a dedicated control device.Source: C. Human Reactions as Temporal… · p. 3

A. Event Representation and Scope

  • The event representation distinguishes fast reactions, extended-context reactions, and direct lexical events, while only binary robot relevance enters the candidate fusion rule.Attention-target estimates remain diagnostic rather than serving as a fusion input.Source: A. Event Representation and Scope · p. 3

B. Audio Reaction Paths

SocialVLA derives causal audio features from emotion-model outputs and uses independent fast and extended-context reaction heads. The longer context can enrich an episode but is not required before an earlier authorized hold.

B. Audio Reaction Paths

  • Audio is recorded as causal 320-ms observations, and seven emotion logits form the acoustic representation for downstream reaction heads.The emotion model is based on wav2vec 2.0.Source: B. Audio Reaction Paths · p. 3
  • The temporal representation uses the latest emotion-logit vector with history statistics and a first-vector difference, producing 28 features; gaps longer than 0.21 s clear the history.The fast head uses up to 0.4 s of history, while the independently trained extended-context head uses up to 2.0 s.Source: B. Audio Reaction Paths · p. 3
  • The fast path targets reactions from the earliest acoustic dynamics, while extended-context estimates capture later-developing prosody and can enrich an open episode without delaying an earlier authorized hold.Audio supervision is modality-local: labels concern intervention-worthy acoustic behavior, not robot-related status or attention targets.Source: B. Audio Reaction Paths · p. 3

C. Direct Lexical Path

SocialVLA pairs an independent lexical stop channel with visual reaction recognition and separately estimated robot relevance. Its soft fusion uses bounded relevance context while allowing the first authorized threshold crossing to open an intervention episode.

C. Direct Lexical Path

  • Vosk independently detects authorized Russian stop commands and negative phrases, contributing a unit-strength lexical candidate alongside learned audio heads for prosodic or nonlexical reactions.Direct phrases can trigger the cascade even when vocal affect is weak.Source: C. Direct Lexical Path · p. 3

D. Video Reaction Head

  • The visual reaction head maps facial-expression, landmark, face-confidence, position, size, and pose features to an intervention-worthy visible-reaction probability.It uses an ExtraTrees classifier; mirrored features augment training partitions but are not added to held-out folds.Source: D. Video Reaction Head · p. 4

E. Robot-Relevance and Attention Heads

  • A separate video-only logistic head estimates robot relevance, while the four-class attention estimate is diagnostic and does not act as a hard stop gate or independent veto.The relevance target comes exclusively from video annotation.Source: E. Robot-Relevance and Attention Heads · p. 4

F. Soft Multimodal Fusion

  • The soft fusion rule combines audio, visual, and lexical reaction evidence with bounded robot-relevance context, and the first authorized threshold crossing opens the intervention episode.Direct lexical events retain a 0.85 relevance-factor floor; relevance can reduce prosodic and facial confidence by at most 30%, and missing relevance defaults to r = 0.5.Source: F. Soft Multimodal Fusion · p. 4

G. Temporal Association and First-Event Intervention

SocialVLA associates recent audio, video, speech, and relevance observations, then requests a hold from the earliest authorized event rather than waiting for complete multimodal agreement.

G. Temporal Association and First-Event Intervention

  • The latest audio, video, Vosk, and relevance observations are associated within W = 700 ms, an event-linking window rather than an additional wait.The interval accommodates natural cross-modal asynchrony and unequal inference times.Source: G. Temporal Association and… · p. 4
  • The first authorized candidate can open a stop episode immediately, with later extended-context and cross-modal evidence attached to that episode.The intervention time is based on the minimum time among authorized fast, extended-context, and direct lexical stop-event classes.Source: G. Temporal Association and… · p. 4
  • Unlike audio-AND-video fusion, first-event fusion need not wait for both modalities and can retain unimodal reactions.Source: G. Temporal Association and… · p. 4

H. Stop-and-Correct Interface

The interface pauses the running VLA until the participant explicitly chooses how to proceed, while synchronized intervention records link reactions and corrections to the active action.

H. Stop-and-Correct Interface

  • One pause command places the VLA in a hold state, and the participant may resume, restart after workspace adjustment, or provide a revised instruction.Further events in the same intervention episode do not issue additional control commands.Source: H. Stop-and-Correct Interface · p. 4
  • A restricted low-latency speech path detects stop phrases during execution, while a separate post-stop session records unrestricted verbal corrections without controlling motion.Only an explicit participant resume command can continue execution.Source: H. Stop-and-Correct Interface · p. 4

I. Reaction-Centered Intervention Records

  • Each intervention record links synchronized reaction, stop-request, and correction times with the active VLA action, contributing modalities, detector evidence, and correction transcript.Source: §I · p. 4
  • These records may support analysis of human-identified problematic execution segments and future policy-data curation, while policy retraining is outside this work's scope.Source: §I · p. 5

IV. EXPERIMENTAL METHODOLOGY

The evaluation tests whether SocialVLA detects an observer's intervention-worthy response and turns it into a timely hold during physical pick-and-place manipulation.

A. Platform and Task

  • Experiments use a physical Unitree G1 controlled by GR00T systems for ball pick-and-place, with participants observing successes and naturally occurring failures without being prompted to react.Source: A. Platform and Task · p. 5
  • The prediction target is an observer's intervention-worthy response, not a predefined taxonomy of physical failure modes.The evaluation asks whether the system recognizes perceived need for intervention and translates it into a timely VLA hold.Source: A. Platform and Task · p. 5

B. Recordings and Annotation

  • Fifteen participants contributed synchronized participant-facing audio-video and robot/model logs; finalized annotations label reaction modalities, intervention status, robot relation, attention, subtype, and linked events.Reaction onset is the first recognizable instant in an isolated short clip, and unannotated time is excluded rather than treated as non-intervention.Source: B. Recordings and Annotation · p. 5
  • The finalized annotations contain 238 robot-related intervention-worthy episodes and 62.3 minutes of explicitly annotated non-intervention behavior.Overlapping audio-video manifestations or those separated by at most 700 ms are merged into one episode.Source: B. Recordings and Annotation · p. 5

C. Evaluation Protocol

  • Frozen full-corpus replay on 15 participants measures channel trade-offs, while prospective deployment on an unseen 16th participant evaluates the complete system without fitting or calibration on that participant.Source: C. Evaluation Protocol · p. 5
  • A prediction matches an episode when its first threshold crossing falls from reaction onset through 1 s afterward; the stricter timely criterion is 500 ms.One-to-one matching prevents a prolonged reaction from counting as multiple true positives, and predictions before reaction onset do not count as detections.Source: C. Evaluation Protocol · p. 5
  • Thresholds are selected on development data to maximize event recall under an exploratory non-intervention false-positive-rate target of 0.10, then frozen for replay and deployment.The 700-ms association interval is a cross-modal linking tolerance, not added detection latency.Source: C. Evaluation Protocol · p. 5

D. Metrics

The metrics cover event predictions and false-stop behavior across non-intervention intervals, with additional measures of delay and timely recall.

D. Metrics

  • Non-intervention false-positive rate counts non-intervention or robot-unrelated intervals containing at least one predicted stop, not individual false-stop episodes.Multiple false-stop episodes can occur within one interval, so this count is not generally the precision false-positive count.Source: D. Metrics · p. 5
  • The evaluation also reports false-stop episodes per hour, median and 95th-percentile reaction-onset delay, and recall within 500 ms.Source: D. Metrics · p. 6

V. EVALUATION RESULTS

The frozen full cascade reached 54.6% event recall and 69.5% precision offline; ablations showed complementary audio and video coverage, while relevance estimation reduced false stops. Prospective timing results show that most delay occurs before detector emission and during physical deceleration.

A. Full Multimodal Offline Evaluation

  • 54.6% event recall and 69.5% precision were achieved by the complete cascade on the 238-episode offline corpus.The replay also found that 45.4% of annotated intervention episodes produced a stop within 500 ms of reaction onset.Source: A. Full Multimodal Offline Evaluation · p. 6

B. Prospective Frozen-System Deployment

  • Median detector-to-fusion and fusion-to-gate delays were 47.9 ms and 0.66 ms, while most end-to-end delay occurred before detector emission and during robot deceleration.Transport remained below 100 ms at p95.Source: B. Prospective Frozen-System Deployment · p. 6

C. Offline Ablation Study

  • 64.3% recall from first-event Audio+Video fusion exceeded Audio-fast recall by 16.0 percentage points, but produced 100 false-stop episodes.The offline ablations used the same frozen configuration across all 15 sessions, without per-ablation retraining or threshold selection.Source: C. Offline Ablation Study · p. 6
  • Adding robot relevance reduced false stops from 100 to 57 and Non-interv. FPR from 41.1% to 27.8%, while precision rose from 60.5% to 69.8%.The relevance-filtered system retained 132 of the 153 true positives found by unfiltered fusion.Source: C. Offline Ablation Study · p. 6
  • The all-session ablation results are descriptive offline comparisons, separate from the prospective real-robot evaluation.Non-interv. FPR uses annotated non-intervention/unrelated episodes; false stops/h uses total non-intervention observation time.Source: C. Offline Ablation Study · p. 6

D. Runtime and Stop–Correct Operation

SocialVLA routes authorized reaction events to a VLA hold, then supports participant-directed continuation, restart, or instruction revision. The discussion emphasizes external reaction evidence’s usefulness and its limits when observers do not react or misinterpret the interaction.

D. Runtime and Stop–Correct Operation

  • Tables I–III cover the full cascade, prospective event detection and latency, and offline ablations under frozen settings, respectively.Table III uses identical frozen weights, thresholds, and a 700-ms cross-modal association window.Source: Table I · p. 7, Table II · p. 7, Table III · p. 7
  • 47.9 ms median and 84.9 ms p95 detector-to-fusion-request delays were measured in the prospective live run; fusion-to-gate delivery took 0.66 ms median and 0.93 ms p95.The 47.6 ms mean offline audio encoding time was measured per 320-ms audio window in batched processing.Source: D. Runtime and Stop–Correct Operation · pp. 6–7
  • The first authorized reaction event sends one stop request; while held, the participant can continue, restart from an adjusted scene, or provide a revised instruction.Later detections from the same reaction do not generate repeated pause commands.Source: D. Runtime and Stop–Correct Operation · p. 7

VI. DISCUSSION

  • Spontaneous human reactions can be converted into an operational runtime signal for an executing VLA without access to policy internals; useful intervention balances reaction coverage, false stops, and delay to physical hold.Source: §VI · p. 7
  • Reaction recognition is separated from robot relevance: audio heads estimate intervention need, while a visual head supplies bounded contextual evidence rather than a hard gate.Fast and extended-context audio heads provide complementary detection horizons; post-stop speech recognition records corrections without affecting initial intervention latency.Source: §VI · pp. 7–8
  • The system is less effective when observers do not react, react ambiguously, or respond to an unrelated stimulus.The discussion recommends combining SocialVLA with robot-internal monitors.Source: §VI · p. 8

VII. LIMITATIONS AND ETHICAL CONSIDERATIONS

The paper reports end-to-end stop-and-correct operation on a physical Unitree G1, including a frozen deployment on an unseen participant, while identifying sample and monitoring-scope boundaries.

VII. LIMITATIONS AND ETHICAL CONSIDERATIONS

  • The evaluation included 15 annotated-corpus participants and one additional prospective participant, which does not establish generalization across diverse demographic and reaction groups.The passage names language, culture, age, appearance, neurodiversity, and reaction style as unestablished dimensions.Source: §VII · p. 8
  • The relevance head may confuse robot-related reactions with coincident behavior, and calibration and false-stop behavior require study on longer recordings with natural distractors.The 0.7 floor limits damage from relevance errors but can also let unrelated strong reactions pass through.Source: §VII · p. 8
  • SocialVLA detects human-perceived intervention need rather than every physical failure, so it is complementary monitoring and not a certified safety system.Hardware emergency stops, conservative robot limits, and independent supervision remain necessary; future work will combine social signals with robot-internal monitors.Source: §VII · p. 8, §VIII · p. 8

VIII. CONCLUSION

  • SocialVLA combines causal audio, visual reaction recognition, stop phrases, and robot relevance to request a hold and capture participant corrections.After a hold, the speech channel supports participant-directed continuation, restart, or instruction revision.Source: §VIII · p. 8
  • The frozen system demonstrated end-to-end stop-and-correct operation on a previously unseen 16th participant.The evaluation also included offline multimodal testing on the 15-participant annotated corpus.Source: §VIII · p. 8
Loading 2610.02360v1…