Source-linked AI summary
ReactMotion: Generating Reactive Listener Motions from Speaker Utterance
Cheng Luo, Bizhu Wu, Bing Li, Jianfeng Ren, Ruibin Bai, Rong Qu, Linlin Shen, Bernard Ghanem
TL;DR
Reactive listener motion generation addresses the underexplored problem of producing appropriate, diverse body responses to speaker utterances. The paper introduces a multi-reaction preference dataset, tier-aware evaluation, and a unified multimodal generator, reporting substantially better motion quality and reactive appropriateness than strong baselines.
Problem
Reactive listener body motions remain overlooked, while one utterance can elicit multiple valid reactions and existing datasets and metrics inadequately capture this one-to-many appropriateness.
Method
The paper constructs ReactMotionNet with multiple preference-tiered reactions per utterance and trains ReactMotion to jointly model transcript, audio, emotion, and motion using preference-based objectives.
Results
ReactMotion substantially outperforms strong baselines in motion quality and reactive appropriateness.
Takeaways & Limitations
Preference-oriented data and evaluation support modeling listener responses beyond a single ground-truth motion.
Takeaways & Limitations
The relatively simple architecture leaves substantial potential for improvement through more advanced architectures and training techniques.
Abstract
from arXiv · showhide
In this paper, we introduce a new task, Reactive Listener Motion Generation from Speaker Utterance, which aims to generate naturalistic listener body motions that appropriately respond to a speaker's utterance. However, modeling such nonverbal listener behaviors remains underexplored and challenging due to the inherently non-deterministic nature of human reactions. To facilitate this task, we present ReactMotionNet, a large-scale dataset that pairs speaker utterances with multiple candidate listener motions annotated with varying degrees of appropriateness. This dataset design explicitly captures the one-to-many nature of listener behavior and provides supervision beyond a single ground-truth motion. Building on this dataset design, we develop preference-oriented evaluation protocols tailored to evaluate reactive appropriateness, where conventional motion metrics focusing on input-motion alignment ignore. We further propose ReactMotion, a unified generative framework that jointly models text, audio, emotion, and motion, and is trained with preference-based objectives to encourage both appropriate and diverse listener responses. Extensive experiments show that ReactMotion outperforms retrieval baselines and cascaded LLM-based pipelines, generating more natural, diverse, and appropriate listener motions.
1 Introduction
The paper introduces reactive listener motion generation as a distinct one-to-many task and proposes a dataset, evaluation protocol, and preference-trained multimodal model to address it.
- Listeners communicate engagement and understanding through posture and subtle gestures, making responsive body motion important for natural dyadic communication.
- Reactive listener motion generation models naturalistic body motions responding to a speaker’s transcript and/or audio, unlike conventional input-content-driven motion generation.
- The task is challenging because one utterance can elicit multiple valid reactions, lacks a large-scale multi-reaction dataset, and requires appropriateness evaluation beyond single-reference metrics.
- ReactMotionNet contains multiple candidate reactions per utterance with Gold, Silver, and Negative preference tiers, capturing one-to-many ambiguity and enabling preference supervision.
- The paper introduces tier-aware ranking evaluation and ReactMotion, a unified model that jointly processes transcript, audio, emotion, and motion with preference-based training.
- The stated contributions are a new task, a multi-tier dataset and evaluation protocol, and a unified multimodal generator for listener responses.
2 Related Work
Prior work generates motions from explicit actions, synchronized signals, actor motion, or video, whereas ReactMotion targets implicit conversational reactions from speaker utterances with one-to-many responses.
- Text- and audio-driven motion methods typically generate gestures or motions aligned with explicit action descriptions or audio signals.
- Existing multi-person motion methods condition on explicit joint-action descriptions or both participants’ audio streams.
- Reactive-motion approaches commonly condition reactor motion on actor motion, including temporal-spatial attention, diffusion models, or facial-expression cues from RGB video.
- Existing interaction datasets often lack speech semantics and emotional cues, or provide mainly upper-body motions in one-to-one speaker-listener pairs.
- ReactMotionNet instead maps utterances to multiple reactions labeled gold, silver, or negative and includes more dynamic body motions.
3 Task Definition
The task learns a conditional distribution over listener reactions given multimodal speaker conditions, reflecting that conversational cues do not explicitly determine a single target motion.
- Given a speaker utterance Cs, the goal is to generate an appropriate reactive body motion Rl for the listener.
- Speaker conditions may contain audio, transcript, emotion, or specified combinations of these modalities.
- The model learns pθ(Rl | Cs), allowing diverse listener reactions to be sampled at inference time.
- Because utterances do not explicitly specify listener motion, the mapping is inherently one-to-many and must balance contextual appropriateness with diversity.
4 ReactMotionNet Dataset
ReactMotionNet is constructed by repurposing motion data into multimodal speaker–listener pairs, filtering and ranking candidates into graded preference tiers.
- Dataset Construction Pipeline: The construction pipeline repurposes existing human motion data into one-to-many speaker utterance–listener reaction mappings without costly data collection.
- Dataset Construction Pipeline: Step 1 curates dynamic listener motions from HumanML3D and filters conversation-irrelevant motions with multiple LLM-based verifiers.
- Dataset Construction Pipeline: Step 2 uses listener-motion captions to infer plausible speaker transcripts and emotions, then synthesizes corresponding audio with TTS.
- Dataset Statistics: The dataset statistics table defines counts for labeled pairs, unique transcripts, audio files, emotion categories, motions, candidates per utterance, and preference labels.
- Dataset Construction Pipeline: Step 3 checks whether synthesized audio reflects its assigned emotion and filters unreliable utterances before pairing candidates.
- Dataset Construction Pipeline: Step 4 scores semantic appropriateness and conversational plausibility, verifies caption inferences with NLI, and assigns Gold, Silver, or Negative tiers.
- ReactMotion Framework: The framework uses modality-specific tokenizers and a Seq2Seq model to unify speaker cues and listener motions, with group-wise preference training for one-to-many behavior.
- Dataset Statistics: ReactMotionNet contains 151,328 labeled pairs from 8,298 utterances and 2,029 motions, averaging 18.24 candidate reactions per utterance.
5 Methodology
ReactMotion unifies multimodal tokenization and autoregressive generation with group-wise preference learning to produce multiple appropriate listener reactions from a speaker utterance. Its training and decoding pipeline represents speech, emotion, and motion in a shared token space, ranks candidate reactions by preference tiers, and reconstructs generated motion from discrete codes.
- Unified multimodal representation: ReactMotion converts speaker transcript, audio, emotion, and listener motion into discrete modality-specific tokens for unified processing.The model uses modality-specific tokenizers and a shared vocabulary containing textual, audio, motion, and special tokens.
- Unified multimodal representation: Audio tokenization preserves prosodic and paralinguistic cues that inform reactive listener behaviors.The audio waveform is encoded, quantized through an audio codebook, and represented as discrete audio tokens.
- Autoregressive generation: Generated motion tokens are mapped to motion-codebook vectors and decoded by a VQ-VAE motion decoder into raw listener motion data.The reverse mapping converts the unified model’s discrete output back into the motion representation used by the application.
- Autoregressive generation: The unified Seq2Seq model autoregressively generates listener motion tokens conditioned on task-template inputs containing the speaker utterance.ReactMotion adopts T5-base and extends its vocabulary with audio and motion code indices, supporting different speaker-input modalities or combinations.
- Group-wise preference learning: Group-wise preference learning samples Gold, Silver, and Negative motions for each utterance and aggregates their length-normalized likelihoods into label-specific scores.The method forms groups from candidate motions, uses smooth log-mean-exp aggregation, and obtains ℓG, ℓS, and ℓN for the three preference tiers.
- Group-wise preference learning: A soft-margin ranking loss encourages the ordering ℓG > ℓS > ℓN, while inverse-frequency weighting reduces dominance from frequently occurring motion sequences.The margin parameter m controls separation between labels, λgn controls the Gold≻Negative constraint, and group weights are based on inverse square-root motion frequency.
6 Experiments
Experiments evaluate reactive appropriateness with tier-aware multimodal judging, alongside motion quality and diversity. ReactMotion outperforms retrieval and cascaded generation baselines, while preference learning and multimodal fusion improve appropriateness-quality trade-offs.
- Evaluation Protocol: The multimodal judge ranks candidate reactions against Gold, Silver, and Negative tiers using win rates and ranking metrics under missing modalities.It supports six input modes and evaluates graded relevance G>S>N.
- Evaluation Protocol: Win(G>N) ≈0.99 and Win(S>N) ≈0.98, while Win(G>S) ≈0.87–0.88, indicating reliable tier separation and sensitivity to fine-grained quality.The judge also achieves MRR(G) ≈0.82–0.84 and nDCG@5 ≈0.87–0.88.
- Quantitative Results: ReactMotion outperforms all baselines in reactive appropriateness, with near-perfect Win(g>N) and substantially improved Win(g>S) and Gen@3.The full T+A+E model has the best overall win rates while maintaining low FID and competitive diversity.
- Qualitative Results: Qualitatively, ReactMotion produces semantically consistent, expressive motions, whereas CE responses are generic and LLM→T2M motions are repetitive.The example contrasts dynamic upper-body and arm movements with a lower-energy Silver reaction under an excited utterance.
- User Study: 67.8% and 72.0% are ReactMotion’s user-study win rates against CE and LLM→MG-MotionLLM, respectively.ReactMotion also receives 44.1% of votes against Silver, versus 31.9% for CE and 31.4% for LLM→MG-MotionLLM.
- Modality and Ablation Studies: Full fusion achieves Win(g>N)=1.000, Win(g>S)=0.797, and FID=4.760, while preference learning improves Win(g>S), Gen@3, and FID over cross-entropy training.The corresponding cross-entropy-to-full-model changes are 0.741→0.797, 0.938→0.960, and 6.555→4.760.
- Modality and Ablation Studies: Removing ranking loss worsens FID from 4.760 to 5.950 while increasing diversity from 4.804 to 5.453.Removing inverse-frequency reweighting causes the largest appropriateness drop, and removing Gold likelihood supervision harms both appropriateness and quality.
7 Conclusion
The paper introduces reactive listener motion generation, ReactMotionNet, preference-oriented evaluation, and ReactMotion, a unified multimodal framework. ReactMotion substantially outperforms strong baselines in motion quality and reactive appropriateness.
- ReactMotionNet captures non-deterministic listener behavior by pairing each speaker utterance with multiple preference-annotated candidate motions.The dataset provides supervision beyond a single ground-truth response.
- Preference-oriented evaluation protocols measure reactive appropriateness for listener motion generation.
- ReactMotion processes multimodal speaker cues and generates listener body motions in response.
- ReactMotion substantially outperforms strong baselines in motion quality and reactive appropriateness.
Outline of the Supplementary Material
The supplementary material documents implementation, model configuration, evaluation, and baseline details. It also provides the organization of these materials by section.
- Section A presents implementation details, including model configuration, vocabulary construction, optimization settings, and training hyperparameters.
- Section A.1 reports ReactMotion’s model size, while Section A.2 provides prompt templates for speaker-condition settings.
- Section B includes multimodal judge formulation, baseline details, and evaluation metrics for reactive appropriateness, motion quality, and diversity.
A Implementation Details
The implementation describes ReactMotion’s multimodal token-based configuration, prompt-controlled input conditions, autoregressive motion-token generation, and a multimodal judge trained for candidate compatibility.
- Implementation Details: ReactMotion uses a T5-base Seq2Seq backbone with 222.9M backbone parameters and 235.9M trainable parameters after vocabulary extension.The vocabulary incorporates motion, audio, and modality-specific tokens.
- Prompt Templates: Speaker transcription, audio, and optional emotion fields are selectively enabled in a fixed prompt template for multiple condition modes.The model outputs only a listener motion-token sequence in a strict format.
- Generation: The model autoregressively predicts the listener motion-token sequence from the constructed speaker prompt and previous output tokens.
- Multimodal Judge: The judge produces a scalar compatibility score in which larger values indicate more appropriate candidate listener motions for the speaker utterance.
- Multimodal Judge: The multimodal judge encodes transcript, audio, emotion, and motion through modality-specific branches and a fusion branch in a shared scoring space.The architecture allows missing modalities and uses attention pooling for hidden representations.
- Preference Training: Training samples candidates from Gold, Silver, and Negative motion tiers and uses a motion bank to discourage overly generic or template-like compatibility judgments.
B.2 Implementation Details of Judge Network
The judge network and evaluation suite compare generated motions with tiered listener reactions using multimodal compatibility scores, ranking metrics, motion quality, and diversity measures.
- Judge Network: The multimodal judge uses transformer-based encoders to project text, audio, emotion, and motion representations into a shared embedding space for compatibility scoring.The implementation uses a T5-base text encoder and transformer processing for other modalities.
- Baseline Methods: The baselines include random selection, transcript-based retrieval, and cascaded LLM-to-T2M pipelines using specified LLM and motion-generation models.
- Evaluation Metrics: Reactive appropriateness compares generated motions against Gold, Silver, and Negative tiers using Win(g>G), Win(g>S), and Win(g>N).Win(g>N) is the least strict comparison, while Win(g>G) is the most challenging.
- Evaluation Metrics: Gen@3 measures whether at least one generated motion ranks within the top three among annotated and generated candidates for the same utterance.This metric reflects the one-to-many nature of plausible listener reactions.
- Evaluation Metrics: Motion quality is measured with Fréchet Inception Distance between generated and real motion feature distributions, with lower values indicating closer distributions.
- Evaluation Metrics: Diversity is measured by the average feature-space distance between two randomly sampled generated-motion subsets, with higher values indicating greater variation.The measure is intended to detect collapse toward repetitive motion patterns.
C More Details of ReactMotionNet Dataset
ReactMotionNet provides large-scale, graded, and one-to-many supervision for reactive listener motion generation, with utterance-disjoint splits supporting generalization evaluation. It covers diverse emotions and multiple candidate reactions per utterance.
- Dataset properties: ReactMotionNet contains over 151K labeled speaker–listener pairs with multiple candidate reactions and Gold, Silver, and Negative supervision.These properties support both generative modeling and preference-aware evaluation.
- Evaluation design: Utterance-disjoint splitting enables cleaner evaluation of generalization to unseen conversational conditions.
- Dataset statistics: The dataset contains 151,328 pairs spanning 8,298 speaker utterances and 2,029 listener motions.Each utterance is paired with 18.24 candidate reactive motions on average.
- Dataset statistics: Gold, Silver, and Negative annotations comprise 9,307, 34,196, and 107,825 pairs, respectively.The 8:1:1 split is performed by speaker utterance, preventing utterance overlap across partitions.
- Emotion coverage: ReactMotionNet covers 47 emotion categories, including amused, angry, anxious, calm, confused, joyful, sad, and surprised reactions.
D.1 Hyperparameter Sensitivity Analysis
The sensitivity analysis identifies moderate preference-loss settings as the most balanced regime, while inference remains efficient and human evaluations favor ReactMotion over both generative baselines. A remaining gap to Silver references indicates unresolved challenges in naturalness, contextual precision, and diversity.
- Hyperparameter sensitivity: A margin of 0.5 improves Win(g>S) from 0.7482 to 0.7966, Win(g>G) from 0.2240 to 0.2663, and Gen@3 from 0.9467 to 0.9600 while reducing FID from 5.2102 to 4.7596.The comparison uses λrank = 0.25 and λgn = 0.25.
- Hyperparameter sensitivity: Increasing λrank from 0.25 to 0.5 and 0.1 decreases Win(g>S) from 0.7966 to 0.7579 and 0.7082 and worsens FID from 4.7596 to 5.3855 and 5.4811.The passage attributes this pattern to excessive ranking pressure reducing generative fidelity.
- Hyperparameter sensitivity: Setting λgn to 1.0 slightly improves pairwise preference scores but lowers Gen@3 and degrades FID compared with λgn = 0.25.The selected main-experiment setting is m = 0.5, λrank = 0.25, and λgn = 0.25.
- Inference efficiency: ReactMotion generates 1.74 motion sequences per second and achieves 1.66 motion sequences per second end to end on a single NVIDIA A100 80GB GPU.The decoder processes 39.1 motion sequences per second, adding minimal computational overhead.
- User study: ReactMotion receives 67.8% preference against CE and 72.0% against LLM→MG-MotionLLM in the user study.Against Silver, ReactMotion receives 44.1% of votes, compared with 31.9% for CE and 31.4% for LLM→MG-MotionLLM.
- User study: The remaining gap between ReactMotion and Silver references leaves room for improvement in motion naturalness, contextual precision, and diversity.
E Limitations
The paper uses a relatively simple architecture to maintain training stability and computational efficiency, while acknowledging substantial room for further improvement. Future work may therefore explore more advanced architectural designs.
- Limitations: The model uses a relatively simple architecture to maintain training stability and computational efficiency.
- Limitations: The authors acknowledge a large potential for further improvement despite the approach’s promising results.
- Limitations: Future work could explore more advanced architectural designs.