PaperlayerView exact revision on arXiv

Source-linked AI summary

MIBURI: Towards Expressive Interactive Gesture Synthesis

M. Hamza Mughal, Rishabh Dabral, Vera Demberg, Christian Theobalt

arXiv:2603.03282v2cs.CVcs.GRcs.HC

TL;DR

Existing ECA systems lack expressive embodied motion, while natural gesture generators typically require future speech and offline runtimes. MIBURI uses Moshi’s internal speech-text tokens, body-part-aware codecs, and a two-dimensional causal generator for real-time gestures and facial expressions. Experiments report natural, contextually aligned gestures that outperform recent baselines, while the current system does not model users’ body dynamics or full dyadic context.

  • Problem

    Existing ECA assistants lack embodied gestures, while expressive generative gesture methods are typically non-causal, require future speech, or have long runtimes.

  • Method

    MIBURI directly conditions a two-dimensional causal gesture generator on Moshi’s internal speech-text tokens and body-part-aware gesture codecs.

  • Results

    MIBURI produces natural, contextually aligned gestures and outperforms recent baselines across single- and multi-speaker settings.

  • Takeaways & Limitations

    MIBURI moves ECAs toward interactive, human-like embodied communication by synchronizing expressive gestures and facial expressions with real-time dialogue.

  • Takeaways & Limitations

    The framework models only the agent’s motion, not the user’s body dynamics or full dyadic context, limiting multi-party gesture interaction.

Abstract

from arXiv · show

Embodied Conversational Agents (ECAs) aim to emulate human face-to-face interaction through speech, gestures, and facial expressions. Current large language model (LLM)-based conversational agents lack embodiment and the expressive gestures essential for natural interaction. Existing solutions for ECAs often produce rigid, low-diversity motions, that are unsuitable for human-like interaction. Alternatively, generative methods for co-speech gesture synthesis yield natural body gestures but depend on future speech context and require long run-times. To bridge this gap, we present MIBURI, the first online, causal framework for generating expressive full-body gestures and facial expressions synchronized with real-time spoken dialogue. We employ body-part aware gesture codecs that encode hierarchical motion details into multi-level discrete tokens. These tokens are then autoregressively generated by a two-dimensional causal framework conditioned on LLM-based speech-text embeddings, modeling both temporal dynamics and part-level motion hierarchy in real time. Further, we introduce auxiliary objectives to encourage expressive and diverse gestures while preventing convergence to static poses. Comparative evaluations demonstrate that our causal and real-time approach produces natural and contextually aligned gestures against recent baselines. We urge the reader to explore demo videos on https://vcai.mpi-inf.mpg.de/projects/MIBURI/.

1. Introduction

MIBURI addresses the gap between expressive but offline gesture synthesis and real-time ECA systems that produce rigid, low-diversity motion. It directly uses speech-text model tokens to generate expressive full-body gestures and facial expressions causally during dialogue.

  • Current conversational assistants lack the body gestures and facial expressions that contribute contextual information to human communication.
  • Real-time rule-based and data-driven systems often produce less expressive, low-diversity motion with artificial turn-taking patterns.
  • Recent generative methods produce natural, expressive gestures but require future speech context and therefore cannot operate in parallel with live speech generation.
  • MIBURI directly taps Moshi’s speech-text token stream to generate expressive co-speech body gestures and facial expressions in an online, fully causal framework.This avoids latency-inducing speech conversion and audio-tokenization stages while using semantic and acoustic context.
  • Its architecture uses two transformers and body-part-aware Residual VQ-VAE codecs to model temporal dynamics and hierarchical kinematic details.Tokens represent short motion windows and separately encode face, upper body, and lower body regions.
  • MIBURI’s experiments analyze design choices and report advances over recent ECA and gesture-synthesis approaches through perceptual and numerical evaluations.

3. Approach

MIBURI combines Moshi’s real-time speech-text tokens with body-part-aware motion codecs and a two-dimensional autoregressive generator. Auxiliary objectives and causal inference mechanisms target expressive, diverse gestures without future speech context or excessive runtime.

  • MIBURI generates full-body gestures and facial expressions synchronized with speech for embodied conversational agents.
  • The framework extracts Moshi’s internal speech-text token stream and autoregressively generates body-region-aware motion tokens.Additional objectives encourage expressive and diverse motion, while efficient attention contexts and caches support causal real-time inference.
  • Moshi jointly autoregressively generates text and speech tokens for full-duplex dialogue, providing semantic and acoustic conditioning features to MIBURI.
  • Body-part wise Gesture Codecs: Body-part-aware codecs separately encode upper body, lower body, and facial motion into discrete tokens for downstream autoregressive generation.Each region uses a separate Residual VQ-VAE codec because articulation scales and speech relationships differ across body regions.
  • Body-part wise Gesture Codecs: Residual VQ-VAE preserves multiple levels of motion detail by quantizing each region’s encoded sequence into hierarchical residual tokens and reconstructing it through a decoder.
  • Body-part wise Gesture Codecs: The concatenated gesture tokens represent temporal dynamics across frames and kinematic detail across body regions at each timestep.
  • Autoregressive & Causal Transformers: A naive autoregressive treatment of all T·K tokens would enlarge attention contexts and make temporal modeling harder to train and compute.
  • Autoregressive & Causal Transformers: The temporal transformer predicts the first kinematic-level token from previous timesteps, while the kinematic transformer predicts subsequent body-part levels using temporal, speech, text, and identity context.

4. Experiments

MIBURI is evaluated perceptually, quantitatively, for latency and causality, and through ablations of its speech encoding, transformer design, losses, and gesture codec. It produces expressive and natural motion against standard baselines, reaches state-of-the-art FGD and BeatAlign in multi-speaker evaluation, and maintains low latency.

  • Perceptual Evaluation: MIBURI’s perceptual evaluation compares its gestures with baseline generations using pair-wise human judgments of naturalness and speech appropriateness.The study uses the BEAT2 test set; quantitative metrics alone may not correlate with human perception.
  • Perceptual Evaluation: MIBURI generates more expressive and natural motion than EMAGE and GestureLSM, but does not match ground-truth quality or speech appropriateness.This limitation is reported from the perceptual user study.
  • Quantitative Evaluation: MIBURI achieves state-of-the-art FGD and BeatAlign in the larger multi-speaker setting without seed sequences or future context.The result indicates benefits from larger, more diverse motion data and scaling across identities.
  • Latency Analysis: 36ms per frame is achieved by the online demo system on an RTX3090, including model runtime and web-dashboard rendering.The low-latency design uses a two-frame token context; render time is excluded from the comparative table’s wall-clock measurements.
  • Ablation Studies: Using Moshi’s internal speech-text token stream improves quantitative metrics and avoids the additional cost of wav2vec audio embedding computation.Wav2vec produces higher FGD and worse BeatAlign than the internal-token configuration.
  • Ablation Studies: A single transformer worsens FID and BeatAlign, lowers diversity, and nearly doubles step time because attention context becomes longer.The two-tier temporal and kinematic design addresses this context-length problem.
  • Ablation Studies: Contrastive loss improves FGD over the base losses, whereas direct MSE on estimated latents increases FGD.The comparison evaluates auxiliary losses alongside cross-entropy and velocity losses.
  • Ablation Studies: Increasing gesture codec levels improves reconstruction quality, and generative FGD follows a similar pattern to MPJPE.MPJPE is used to evaluate reconstruction quality across codebook levels.

5. Limitations & Future Work

MIBURI currently models only the agent’s motion, excluding the user’s body dynamics and full dyadic context, which limits interactive multi-party gesture handling.

  • MIBURI models only the agent’s motion and excludes the user’s body dynamics and full dyadic context.
  • This scope limits the framework’s ability to handle interactive, multi-party gestures.Extending MIBURI to perceive and respond to a partner’s gestures is identified as future work.

6. Conclusion

MIBURI is an online, causal framework for synchronizing expressive co-speech gestures and facial expressions with real-time dialogue. Its body-part-aware codecs, two-dimensional causal generator, and contrastive objectives support low-latency, diverse motion that outperforms recent baselines.

  • MIBURI generates expressive co-speech gestures and facial expressions synchronized with real-time dialogue.
  • Body-part-aware gesture codecs and a two-dimensional causal generator model temporal and kinematic motion structure at low latency.
  • Contrastive objectives enhance gesture diversity and expressiveness.
  • Experiments across single- and multi-speaker settings show natural, contextually aligned gestures that outperform recent baselines.
  • The approach moves ECAs closer to interactive, human-like embodied communication.

7. Online Generation Demo

The online demo showcases MIBURI generating gestures and speech during fully interactive, full-duplex conversation. Three concurrent processes exchange streamed data through websockets to maintain synchronized, low-latency interaction and visualization.

  • The demo showcases online, continuous, and responsive gesture generation during live ECA interaction.
  • Full-duplex interaction lets users and the ECA speak, interrupt, and respond fluidly instead of following traditional turn-based exchanges.
  • Three parallel processes run inference, speech/text visualization, and motion visualization concurrently.They communicate through lightweight websocket channels for synchronized, low-overhead data exchange.
  • The combined components enable seamless, continuous interaction while maintaining low latency at each stage.
  • The architecture streams data between processes at each timestep via websockets to support low-latency, full-duplex interaction.The main inference loop runs Moshi and MIBURI continuously while visualization processes operate in parallel.

8. Additional Results on Embody3D [33]

On the 59-hour Embody3D dyadic-interaction dataset, MIBURI achieves the best quantitative results across FGD, BeatAlign, and L1 Divergence. The findings indicate lower distribution divergence, stronger prosodic alignment, and motion diversity closer to ground truth in a new conversational setting.

  • Embody3D contains 59 hours of dyadic interaction recordings in which two interlocutors communicate naturally face to face.
  • The dataset evaluates MIBURI in a conversational setting aligned with the goal of human-like interactive embodied agents.
  • Across FGD, BeatAlign, and L1 Divergence, MIBURI achieves the best quantitative results on Embody3D.
  • The results show lower gesture distribution divergence, improved alignment with speech prosody, and motion diversity closer to ground truth.
  • The performance trend mirrors the BEAT2 multi-speaker evaluation, demonstrating generalization to new conversational settings.

9. Analyzing Autoregressive Dependency in Kinematic Transformer.

MIBURI’s autoregressive ordering creates potential dependencies among upper-body, lower-body, and face tokens. Attention analysis shows that face prediction implicitly ignores lower-body tokens despite their earlier prediction order.

  • The causal transformer predicts body parts in an order that makes later lower-body and face predictions depend on earlier upper-body predictions.
  • Figure 6 represents kinematic dependencies with arrows indicating which tokens attend to others.
  • Face tokens implicitly ignore lower-body tokens even though lower-body tokens are predicted earlier.

10. On Causality-Quality Trade-off

MIBURI’s interactive design requires both causality and real-time processing, creating a quality trade-off. Causal modeling favors readily discoverable temporal correlations, whereas future-context models capture more nuanced gestures.

  • Interactive embodied agents require processing that is both causal and real-time, shaping MIBURI’s design choices.
  • Causal modeling favors prominent beat gestures because speech–gesture temporal correlations are easier to discover during training.
  • Non-causal full-context models can represent more nuanced and semantically meaningful gestures by using future speech context.
  • Jointly modeling gesture intent before speech and gestures would break causality and be slow, so intent modeling remains future work.

11. Implementation Details of Gesture Codecs.

The gesture codecs use streaming Residual VQ-VAE architectures to compress and reconstruct motion, with body-specific networks and reconstruction, geometric, and commitment losses. Evaluation uses complementary motion quality, synchronization, diversity, facial accuracy, and pose-accuracy metrics.

  • Streaming Residual VQ-VAE codecs downsample motion by 2 and reconstruct sequences through decoder upsampling.A 250-frame training input produces 125 tokens for a 10-second sequence.
  • Upper- and lower-body codecs use two convolutional and eight transformer layers, while the face codec uses two convolutional and four transformer layers.The upper and lower body use four attention heads; the face uses two.
  • Codec training combines reconstruction, geometric, and commitment losses, including geodesic rotation loss and MSE for multiple motion representations.
  • Fréchet Gesture Distance measures generated-gesture embedding quality using the BEAT2 gesture encoder without retraining it.
  • Beat Alignment Score measures gesture–speech prosodic synchronization, while L1 Divergence detects collapse toward static poses.
  • Facial-MSE evaluates facial-expression error in FLAME representation, and MPJPE measures mean 3D joint-position error across joints and frames.

13. Details on User Study

The user study evaluates perceptual quality through forced-choice comparisons between MIBURI, established baselines, and ground truth. Baselines are selected or retrained to support single-speaker, multi-speaker, and causal comparisons.

  • The perceptual study involved 53 participants, each answering 15 randomly sampled forced-choice questions from 45 total questions.
  • Participants compared side-by-side animations of MIBURI against EMAGE, GestureLSM, or ground truth.
  • Single-speaker evaluations used released baseline checkpoints, while multi-speaker evaluations retrained EMAGE and MambaTalk on a 23-speaker BEAT2 subset.
  • Causal GestureLSM and MambaTalk variants used causal attention masks to enable quantitative comparisons under equivalent online constraints.
Loading 2603.03282v2…