Source-linked AI summary
Multimodal Voice Activity Projection for Social Robot Mediation: Expected Behavior and Deployment Constraints
Antonio Cano, Guillermo Perez, Luis Merino, Randy Gomez
TL;DR
Social-robot mediators need to anticipate how the conversational floor evolves, rather than react only to utterance endings. This paper proposes MM-VAP, which projects floor activity from synchronized audio-visual input and infers turn-taking events. Results support feasibility on NoXi, NoXi+J, and Haru EDR, while real-time deployment remains an open challenge.
Problem
Reactive end-of-turn decisions based on silence or voice activity are limited when speakers pause within turns, backchannel, overlap, or signal transitions before silence.
Source: §I · p. 1Method
MM-VAP uses synchronized audio-visual inputs and multimodal encoding to estimate future floor activity and provide event predictions for a future robot behavior layer.
Source: §I · p. 1, §III · p. 2Results
VA-related pretrained backbones provide a clear advantage over the CPC+3DResNet baseline on NoXi, and NoXi+J and Haru EDR results support feasibility for floor-management events.
Source: §V · p. 4Takeaways & Limitations
The predicted events can inform gaze toward current or expected speakers, while backchannel prediction can support suppressing intervention and floor estimates can feed a dialogue manager’s mediation choices.
Source: §VI · p. 4No similar multimodal turn-taking projection system has yet been demonstrated online in a physical robot, and real-time inference, preprocessing, synchronization, and robustness remain deployment constraints.
Source: §IV · p. 3, §VI · p. 4
Abstract
from arXiv · showhide
Turn-taking prediction is especially relevant for social robots that act as mediators in human-human interaction, where the expected action is often not to speak, but to orient, wait, avoid interruption, or prepare a balanced intervention. This paper presents Multimodal Voice Activity Projection (MM-VAP) as a human state-aware perception layer for future robot mediation behavior. The model estimates the future evolution of the conversational floor from synchronized audio-visual evidence and derives turn-taking events such as Hold, Shift, Shift prediction, Backchannel prediction, and overlap-related states. The approach uses VA-related pretrained audio-visual encoders, LoRA adaptation, inter-speaker attention, and zero-shot event inference from future voice activity projections. Experiments on NoXi, NoXi+J, and Haru EDR support the feasibility of this formulation, especially for floor management events that can be connected to gaze preparation, active listening, and conservative intervention. Finally, the paper defines the expected robot output interface and discusses the main deployment constraints, including real-time inference, preprocessing latency, multimodal synchronization, and input-quality monitoring.
I. INTRODUCTION
Turn-taking prediction matters for mediating human-human dialogue because a robot must track how the conversational floor evolves, not merely detect utterance endings. The paper frames MM-VAP as a human-state perception layer for future mediation behavior.
I. INTRODUCTION
- Reactive end-of-turn decisions based on silence or voice activity can miss within-turn pauses, backchannels, overlaps, and transitions before silence.An incorrect intervention may interrupt a speaker, reinforce overlap, or reduce dialogue naturalness.Source: §I · p. 1
- Human turn-taking is predictive: participants estimate whether the floor will be held, shifted, closed, or briefly supported by listener feedback.Speech production takes longer than typical turn-transition gaps, so responses must be prepared before the current turn is complete.Source: §I · p. 1
- The work builds on MM-VAP for Haru mediation and presents a predictive audio-visual model, an event-to-output interface, and deployment constraints.Haru is framed as an embodied mediator that observes dyadic exchange and may regulate flow, balance participation, manage silences, and express active listening.Source: §I · p. 1
II. RELATED WORK
Prior turn-taking systems commonly detect speech endings reactively, while VAP projects future voice activity to support earlier event prediction. Multimodal extensions add non-verbal evidence, but their visual features and cross-modal learning remain constrained.
II. RELATED WORK
- Classical systems detect speech endings from acoustic activity and silence, whereas predictive turn-taking models estimate future voice activity at frame level.VAP predicts the future binary activity of two speakers and derives events such as shift, hold, backchannel, and end-of-turn from future-state probabilities [7].Source: §II · p. 1
- Multimodal VAP extensions add gaze, facial features, head pose, action units, or visual encoders as evidence for conversational-event prediction – [11].The cited extensions include backchannel prediction, multilingual settings, and real-time implementations.Source: §II · p. 2
- Many approaches use engineered descriptors or visual backbones not pretrained for voice-activity dynamics, while independent modality encoding leaves cross-modal relations to limited VAP training data.Source: §II · p. 2
III. PROPOSED METHOD: MULTIMODAL VOICE ACTIVITY PROJECTION (MM-VAP)
MM-VAP projects the joint future voice activity of two interlocutors from synchronized audio-visual context, then infers turn-taking events from the projected state distribution. Its pipeline combines pretrained multimodal encoding, inter-speaker attention, complementary prediction heads, and a loss spanning activity, projection, and semantic regularization.
III. PROPOSED METHOD: MULTIMODAL VOICE ACTIVITY PROJECTION (MM-VAP)
- The joint future voice-activity projection spans a finite horizon and represents both speakers’ expected activity trajectory in 256 possible states.Four heterogeneous temporal bins per speaker cover immediate, near, and longer-term future regions.Source: §III · p. 2
- Synchronized speaker-specific audio and face sequences are encoded with modality branches and fused using the backbone’s original inter-modality attention.TalkNet and WhisperFlamingo are candidates because their pretrained tasks relate to voice activity and audio-visual speech understanding.Source: §III · p. 2
- LoRA redirects pretrained backbone representations toward future projection while freezing original weights and training low-rank matrices in selected layers.This reduces trainable parameters while preserving the backbone structure and pretrained capabilities.Source: §III · p. 2
- Inter-speaker cross-attention conditions predictions on both participants, and separate heads estimate future VAP states and each speaker’s binary voice activity.Aggregating probability mass over associated projected states yields zero-shot Shift, Hold, Shift prediction, Backchannel prediction, and overlap-related events.Source: §III · p. 3
- The training objective combines binary voice-activity loss, cross-entropy over 256 future states, and semantic grouping of equivalent states.Semantic grouping reduces the influence of redundant microstates.Source: §III · p. 3
IV. MM-VAP FOR HARU MEDIATION: EXPECTED OUTPUTS AND DEPLOYMENT CONSTRAINTS
The paper treats MM-VAP as a perception layer whose event predictions can inform gaze, waiting, and mediation decisions rather than directly determine the robot’s full strategy. It proposes event-specific gaze behavior and identifies signal quality and real-time operation as deployment constraints.
IV. MM-VAP FOR HARU MEDIATION: EXPECTED OUTPUTS AND DEPLOYMENT CONSTRAINTS
- MM-VAP provides event-level estimates of current and future floor state that a robot controller can translate into gaze, waiting, or mediation actions.The formulation estimates who is speaking, who may speak next, and whether the floor is stable, changing, overlapping, or supported by feedback.Source: §IV · p. 3
- Hold supports maintaining gaze on the active speaker, Shift prepares gaze toward the expected next speaker, and overlap shifts gaze toward the participant taking or contesting the floor.Backchannel predictions preserve the current floor configuration and suppress intervention.Source: §IV · p. 3
- Mediation choices depend on dialogue history: after silence, dominance, or imbalance, the robot may orient toward the less talkative participant before prompting or yielding the floor.Longer-term variables may include speaking time, floor acquisitions, silence duration, and overlap frequency; the controller chooses among gaze, waiting, suppression, and mediation actions.Source: §IV · p. 3
- Real-time physical-robot deployment remains un demonstrated, with parallel acquisition, tracking, processing, synchronization, filtering, and control potentially straining latency, memory, and robustness.Missing faces, tracking instability, noise, occlusion, or off-camera speakers can reduce prediction reliability, motivating conservative behavior when inputs are degraded.Source: §IV · p. 3
V. EXPERIMENTS AND RESULTS
Experiments evaluate future voice-activity projections on NoXi, NoXi+J, and Haru EDR. VA-related pretrained backbones outperform the CPC+3DResNet baseline on NoXi, and the Haru EDR results suggest the model captures mediation-relevant floor events.
V. EXPERIMENTS AND RESULTS
- The protocol trains on synchronized audio-visual context windows to predict 2 s of future voice activity, evaluating floor-development events with accuracy and F1-score.NoXi and NoXi+J contain multilingual expert-novice dyadic interactions; Haru EDR covers semi-structured exchanges in Haru’s presence.Source: §V · p. 4
- WhisperFlamingo reached 0.97 F1 for both S/H and S-pred on NoXi; TalkNet reached 0.94 F1 for S/H and 0.93 F1 for S/L, while the pretrained backbones advantage the CPC+3DResNet baseline.On complete NoXi+J, TalkNet had the strongest reported BC-pred score and WhisperFlamingo the strongest S-pred score.Source: §V · p. 4, Table I · p. 4
- WhisperFlamingo reached 0.92 F1 for Hold, 0.85 F1 for Shift, and 0.91 F1 for S-pred on Haru EDR; TalkNet reached 0.84, 0.79, and 0.87, respectively.The evaluation was restricted to mediation-related events; these results suggest the model captures floor maintenance and transitions relevant to mediation.Source: §V · p. 4, Table II · p. 4
VI. DISCUSSION AND CONCLUSIONS
The discussion frames MM-VAP as a human state-aware perception layer for social robot mediation, not a complete behavior policy. Results support feasibility, while deployment and user validation remain open.
VI. DISCUSSION AND CONCLUSIONS
- Results on NoXi, NoXi+J, and Haru EDR support the feasibility of VA-related pretrained audio-visual encoders for MM-VAP, but real-time deployment remains the central open problem.Online inference, lightweight adaptation, behavior-level validation, and integration with Haru’s multimodal dialogue architecture are future work.Source: §VI · p. 4
- Decisions about when to verbally mediate or intervene in overlap are left to additional rules and future user studies, rather than the model alone.The model is intended to estimate the conversational floor and provide a small set of candidate outputs, not decide the robot’s complete social strategy.Source: §VI · p. 4
- Whether predictive floor-state estimation improves human understanding and reduces inappropriate interventions relative to a reactive end-of-speech baseline remains a practical question for future testing.The proposed sequence is to test first in simple 1:1 human-robot interaction, then in mediated human-human interaction.Source: §VI · p. 4
- MM-VAP estimates conversational-floor states for a dialogue manager to use in deciding when to observe, prepare, or mediate, rather than replacing the manager.The paper maps predicted events to candidate outputs such as gaze preparation, waiting, and conservative intervention; the interface requires user validation before serving as a complete mediation policy.Source: §VI · p. 4