Source-linked AI summary
DyaDiT: A Multi-Modal Diffusion Transformer for Socially Favorable Dyadic Gesture Generation
Yichen Peng, Jyun-Ting Song, Siyeol Jung, Ruofan Liu, Haiyang Liu, Xuangeng Chu, Ruicong Liu, Erwin Wu, Hideki Koike, Kris Kitani
TL;DR
Existing gesture-generation methods commonly focus on single-speaker audio and underrepresent social context and mutual dyadic dynamics. DyaDiT is a multimodal diffusion transformer that combines dyadic audio with optional social attributes and partner motion, using ORCA and a motion dictionary. It outperforms prior methods on standard metrics, is preferred by users, and remains limited by dataset constraints and conditioning conflicts.
Problem
Existing methods often map single-speaker audio to motion without explicitly modeling social context or the mutual dynamics of dyadic conversation.
Method
DyaDiT generates dyadic gestures using multimodal diffusion-transformer conditioning on audio, social attributes, partner motion, ORCA, and motion-dictionary priors.
Results
DyaDiT consistently outperforms existing methods on standard metrics and is preferred by users for perceived realism and social consistency.
Takeaways & Limitations
DyaDiT supports socially consistent and natural conversational behavior by capturing interpersonal dynamics between two speakers.
Takeaways & Limitations
The model is limited by its dataset and may suffer conditioning conflicts because personality cues can be implicitly encoded in audio, reducing controllability and diversity.
Abstract
from arXiv · showhide
Generating realistic conversational gestures are essential for achieving natural, socially engaging interactions with digital humans. However, existing methods typically map a single audio stream to a single speaker's motion, without considering social context or modeling the mutual dynamics between two people engaging in conversation. We present DyaDiT, a multi-modal diffusion transformer that generates contextually appropriate human motion from dyadic audio signals. Trained on Seamless Interaction Dataset, DyaDiT takes dyadic audio with optional social-context tokens to produce context-appropriate motion. It fuses information from both speakers to capture interaction dynamics, uses a motion dictionary to encode motion priors, and can optionally utilize the conversational partner's gestures to produce more responsive motion. We evaluate DyaDiT on standard motion generation metrics and conduct quantitative user studies, demonstrating that it not only surpasses existing methods on objective metrics but is also strongly preferred by users, highlighting its robustness and socially favorable motion generation. Code and models will be released upon acceptance.
1. Introduction
Natural digital-human interaction requires gestures that evolve with conversation, but existing models often overlook social context and dyadic interaction dynamics. DyaDiT addresses this gap with socially conditioned dyadic gesture generation and reports stronger quantitative and user-preference results.
- Digital humans need conversational gestures alongside speech to support natural, socially engaging interaction.
- Existing gesture models often omit personality, relationships, conversational roles, and other social factors, producing generic or unnatural motion.
- DyaDiT is a diffusion-based transformer that generates socially aware gestures from dyadic audio while conditioning on explicit relationship and personality cues.
- ORCA disambiguates overlapping speaker audio streams, while optional partner-motion conditioning supports more coordinated and responsive gestures.
- DyaDiT consistently outperforms existing methods on standard metrics, and users prefer its gestures for perceived realism and social consistency.
2. Related Work
Prior work has focused largely on single-speaker co-speech gestures, while dyadic generation requires modeling both participants and their interpersonal dynamics. DyaDiT builds on diffusion modeling and uses multimodal inputs, explicit social conditioning, and audio disentanglement to address this setting.
- 2.1. Co-Speech Gesture Generation: Most earlier co-speech gesture methods synthesize movement for a single speaker from speech-related cues.
- 2.2. Dyadic Gesture and Reaction Generation: Dyadic gesture generation remains largely unexplored because it must model both participants’ behaviors and interpersonal dynamics.
- 2.3. Diffusion-Based Gesture Generation: DyaDiT combines audio, partner motion, relationship type, personality scores, ORCA, and a motion dictionary for style-aware dyadic gesture generation.
- 2.2. Dyadic Gesture and Reaction Generation: Prior dyadic approaches often overlook social context or treat two-speaker audio as one blended signal, obscuring roles and interaction patterns.
- 2.3. Diffusion-Based Gesture Generation: Diffusion models are adopted for gesture generation because dyadic conversation involves multimodal variability and nondeterministic interaction dynamics.
3. DyaDiT
DyaDiT generates upper-body gestures from dyadic audio while conditioning on social attributes and conversational context. Its ORCA module separates complementary speaker cues, while motion dictionaries, partner gestures, and latent motion tokens add interaction and style information.
- Model overview: DyaDiT generates plausible upper-body gestures for one conversational partner from both speakers’ audio and optional social and motion conditions.Its conditions include relationship type, personality, and the partner’s gesture sequence.
- Audio fusion: ORCA filters redundant information between the two audio streams and exchanges complementary cues through bidirectional cross-attention.A learnable gate adaptively fuses the two cross-attention outputs into the final audio conditioning.
- Audio fusion: The fused audio representation reflects both interlocutors’ vocal behaviors and supports generating either speaker or listener gestures.Figure 3 describes natural motion adjustment when one person interrupts the other.
- Motion dictionary: The motion dictionary uses learnable motion bases and style-dependent weights to modulate partner audio features according to current motion style.At inference, style conditioning can be strengthened, omitted, or optionally activated.
- Motion tokenizer: A residual VQ-VAE converts continuous gestures into compact quantized latent tokens for diffusion training and reconstructs continuous motion during inference.The latent representation has temporal length T/4 and dimension d = 64 in the reported experiment.
- Evaluation: Table 1 compares DyaDiT and baselines using Fréchet Distance and Diversity, where lower FD indicates higher realism and higher Diversity indicates more varied motion.The reported comparison also includes references for dataset variation and cross-sample mismatch.
4. Experiments
DyaDiT is evaluated against adapted dyadic gesture-generation baselines using distribution and diversity metrics, with ablations testing its major components. It achieves the strongest overall performance, while ORCA, discrete motion priors, and social conditioning contribute to generation quality.
- Quantitative Evaluation: The evaluation compares DyaDiT with ConvoFusion and Audio2PhotoReal using Fréchet Distance (FD) and Diversity metrics.Ground-truth references provide motion-distribution and variation baselines.
- Quantitative Results: DyaDiT achieves the best overall performance while maintaining strong motion diversity.Table 1 also includes GT diversity as an upper-bound variation reference and GT Random FD as a lower-bound mismatch reference.
- Ablation Studies: Removing ORCA degrades performance, especially in FD, supporting its role in improving motion realism.The ablation compares raw audio concatenation and standard cross-attention alternatives with the full model.
- Ablation Studies: Replacing the discrete Motion Dictionary with a continuous representation reduces performance, particularly in Diversity (Static).The result indicates that discrete motion bases better capture diverse interaction styles.
- Ablation Studies: Mismatched or absent social conditioning reduces Diversity (Static), while the fully conditioned model produces richer gestures in dyadic listener segments.The full model outperforms both Uncond and Random social-conditioning variants in overall generation quality.
5. User Study
The user study compares DyaDiT with ConvoFusion and ground-truth motion across gesture quality, relationship consistency, and personality consistency. Participants preferred DyaDiT, including slight preference over ground truth in some settings.
- Study Design: Sixteen participants evaluated paired clips for overall quality, relationship consistency, and personality consistency.The study used randomized video-pair order to reduce ordering bias.
- Study Design: The study sampled 56 ten-second validation sequences and asked participants to judge human-likeness, relationship fit, and personality fit.Personality questions covered agreeable, conscientious, extraverted, and neurotic traits.
- Results: DyaDiT achieved higher user preference scores than ConvoFusion and slightly outperformed ground-truth gestures.Figure 7 reports A/B preferences against both comparison conditions.
- Results: DyaDiT was preferred by 73.9% for overall quality, 69.8% for relationship consistency, and 66.7% for personality consistency.These results support higher perceived quality and social appropriateness in the evaluated comparisons.
- Statistical Analysis: Quality and personality preferences significantly deviated from chance, while personality preference also exceeded ground truth.The reported tests were χ2, p < 10−8 against baseline and χ2, p = 1.5×10−4 against ground truth for personality.
6. Conclusion
DyaDiT generates socially consistent dyadic gestures by jointly modeling audio, social attributes, and partner motion. Its ORCA module and motion dictionary support speaker-specific audio interpretation and expressive, style-aware behavior, with evaluations showing superior realism, diversity, and social coherence.
- DyaDiT jointly conditions on dyadic audio, social attributes, and partner motion to model interpersonal dynamics between two speakers.
- ORCA clarifies each speaker’s audio contribution, while the motion dictionary provides style-aware motion priors for expressive generation.
- Quantitative and subjective evaluations show superior performance over existing methods in realism, diversity, and social coherence.
7. Limitation & Future Work
DyaDiT remains constrained by its dataset and may experience conditioning conflicts when personality information is implicitly encoded in audio. Future work targets better disentanglement and broader full-body dyadic data.
- The model is limited by its dataset and may suffer conditioning conflicts because personality cues can be implicitly encoded in audio.
- Implicit personality encoding in audio can reduce controllability and diversity.
- Future work will explore audio neutralization and collect larger dyadic gesture datasets for full-body motion generation.
Gesture Generation
The supplementary materials provide implementation resources, evaluation details, and qualitative demonstrations for DyaDiT. They include clustering analyses, technical documentation, an A/B-test questionnaire, a narration video, code, and planned model release information.
- The supplement includes sections on relationship and personality clustering, DyaDiT implementation details, and the A/B-test questionnaire.
- A narration video provides a paper overview and qualitative gesture-generation examples.
- The authors provide implementation code and state that trained models will be released upon acceptance.
9. Clustering of Generated Gestures
The clustering analysis examines how generated gestures vary with relationship and personality conditioning. Personality produces clearly separated clusters, whereas relationship categories form a more continuous, overlapping manifold.
- The analysis generates t-SNE embeddings under varying relationship and personality conditioning signals.Relationship types are varied while personality scores are fixed; personality features are discretized into five one-hot vectors.
- Personality clusters form clearly separable groups, indicating captured global behavioral tendencies associated with different traits.
- Relationship clusters are less separated because Friend, Family, and Dating gesture styles overlap, producing a continuous manifold.
10. Implementation Details
DyaDiT uses multimodal audio, motion, and social-context representations to generate upper-body dyadic gestures, while limiting supervision to more reliable annotations and joints.
- Audio Processing: Two independent Wav2Vec2 processors extract features from both speakers’ conversational speech for multimodal gesture generation.The resulting features are projected, normalized, and fused to combine self and other speaker cues.
- Context and Motion Priors: A learnable motion bank provides 1000 prototype vectors in R512 as contextual motion priors through cross-attention.Relationship and personality embeddings are also projected into R512 and injected into DiT blocks.
- Motion Tokenization: The motion tokenizer uses temporal VQ-VAE compression and residual vector quantization to produce compact 64-dimensional latent embeddings.Pose sequences are downsampled by a factor of 4 before quantization and reconstructed to the original temporal resolution.
- Dataset: Experiments use approximately 182 hours of naturalistic interactions and 3000 paired motion-audio samples from the Seamless Interaction Dataset.Training uses the first 10 official training archives, while testing uses the first archive from the official test split.
- Data Processing: The model retains only 43 upper-body joints because the dataset’s lower-body estimates contain noticeable inaccuracies from limited views and occlusions.Unused joints, global orientation, and root translation are set to zero for visualization.
- Social Annotations: The study uses relationship and personality cues instead of IPC supervision because the available IPC annotations are noisy and ambiguously assigned to speakers.The authors plan to add IPC-aware conditioning after future annotation refinement.
- Limitations: Future work plans to re-annotate video with advanced pose-estimation tools to obtain more reliable full-body motion supervision.This identifies full-body supervision reliability as a current data limitation.
11. Questionnaire
The questionnaire evaluates gesture quality, relationship consistency, and personality consistency through paired video comparisons under two evaluation settings.
- Questionnaire Design: The questionnaire contains 28×2 questions covering overall gesture quality, relationship consistency, and personality consistency.The three categories contain 10, 8, and 10 questions, respectively.
- Questionnaire Design: Each question presents paired gesture videos comparing DyaDiT with ConvoFusion or Ground Truth.These comparisons support preference evaluation across both a baseline and reference condition.
- Interface: The reconstructed questionnaire interface lets reviewers browse every question and play the corresponding videos.The original study was conducted through Google Forms.
- Interface: Figure 9 shows the Google Forms questionnaire used for the user study.The questionnaire evaluates paired gesture videos through the listed comparison settings.