Source-linked AI summary
DynaConTalk: Wavelet-Constrained Diffusion for Long-Form and Controllable Holistic Co-Speech 3D Motion
Yifei Zhu, Yangyang Cai, Mingyi Shi, Miao Cheng, Lin Gu, Taku Komura, Yoshifumi Kitamura
TL;DR
Holistic co-speech generation can average away layered motion detail, while static fusion can let dense rhythmic cues weaken sparse semantic gestures. DynaConTalk combines aligned SWT-domain diffusion, adaptive speech conditioning, and matched-noise constraints; experiments support improved generation quality and associate controlled analyses with dynamic shape, activity calibration, smoother transitions, and localized control.
Problem
Coordinate-space diffusion entangles posture, gestures, and fine detail, while fixed fusion can let dense rhythmic cues weaken or replace sparse semantic gestures.
Source: §1 · pp. 1–2Method
DynaConTalk uses aligned SWT-domain diffusion, dynamic state-aware speech conditioning, and matched-noise constraint injection for holistic motion generation and control.
Source: §1 · p. 2Results
Experiments support improved generation quality, while controlled analyses associate dynamic shape with SWT, activity calibration with DGN, and smoother transitions and localized control with matched-noise constraints.
Source: §6 · p. 12Takeaways & Limitations
The controlled analyses associate distinct roles with the representation, conditioning, and constraints: dynamic shape, activity calibration, and smoother transitions with localized control, respectively.
Source: §6 · p. 12Datasets provide limited global locomotion, leaving root-trajectory generation weakly constrained by speech alone; generalization to diverse speakers and locomotion-heavy sequences remains unevaluated.
Source: §5 · p. 11
Abstract
from arXiv · showhide
Holistic co-speech animation is prone to averaging in both motion representation and speech conditioning. In coordinate-space diffusion, slow body posture, mid-frequency gesture strokes, and fast hand or facial details are entangled in one prediction target, often producing low-variance, over-smoothed motion. Meanwhile, dense rhythmic and acoustic cues can dominate sparse content-specific information under fixed multimodal fusion. We present DynaConTalk, a wavelet-constrained diffusion framework for long-form and controllable holistic co-speech motion generation. Diffusion operates in stationary wavelet transform (SWT) coefficient space, whose temporally aligned bands separate coarse posture evolution, gesture strokes, and fine expressive details. Our dynamic gating network preserves HuBERT and speaker identity as a base and selectively adds rhythm, mel, and transcript features through motion-state- and noise-aware residual gates. Attention pooling and learned depth routing deliver complementary conditions to each denoising stage, while a frame-resolution rhythm path preserves precise timing. A signed proposal-consensus update then reconciles these conditions with the evolving motion state. Matched-noise constraint injection uses the same sampling interface for history continuation and localized keypose repair, and extends to reference-guided control. Separate body-hand and facial denoisers, followed by inverse SWT and a pose-driven root regressor, produce holistic motion. Experiments evaluate generation quality, facial accuracy, temporal continuity, and controllable editing. Code, models, and the interactive editing interface are available at https://github.com/zhuyifeiabcd1/DynaConTalk.
1 Introduction
Holistic co-speech motion can average away expressive detail because motion scales are entangled and fixed speech fusion can let dense rhythmic cues overwhelm sparse semantic cues. DynaConTalk addresses these problems with wavelet-structured motion diffusion, adaptive speech conditioning, and matched-noise constraints for long-form generation and control.
Problem
- Layered posture, gesture, and expressive detail share one coordinate-space target, encouraging safe, over-smoothed motion that absorbs rare expressive gestures into common beat-like movements.Fixed speech fusion can also let frequent rhythmic cues weaken or replace rare semantic gestures with generic motion.Source: §1 · pp. 1–2
Motion representation
- DynaConTalk diffuses in aligned SWT space, separating coarse posture, mid-frequency gestures, and fine expressive detail for scale-specific learning.Dedicated prediction heads prevent low-amplitude expressive dynamics from being overwhelmed by dominant coarse motion.Source: §1 · p. 2
Speech conditioning
- Dynamic conditioning keeps HuBERT and speaker identity as a base while gating rhythm, mel, and transcript cues using noisy motion and diffusion timestep.Attention pooling, depth routing, frame-level rhythm, and signed proposal–consensus updates adapt conditions to denoising stages and motion state.Source: §1 · p. 2
Long-form generation and control
- Matched-noise constraint injection provides one sampling interface for history continuation, localized keypose repair, and reference-guided control.Clean history or reference targets are re-noised to the current diffusion timestep before injection into constrained regions.Source: §1 · p. 2
2 Related Work
Prior work progressed from authored gesture pipelines to regression and generative models, while co-speech systems increasingly combine heterogeneous speech cues and holistic motion. DynaConTalk is distinguished by dynamically regulating speech conditions and using temporally aligned multiscale motion representation for generation and control.
2.1 From Neural Regression to Holistic Diffusion
- Direct speech-to-motion regression with convolutional and recurrent models tends to average over the one-to-many mapping between speech and valid gestures; recent systems expanded synthesis from upper-body gesture to coordinated body, hands, face, and trajectory [Chen et al. 2024b; Liu et al. 2024b,a; Ng et al. 2021, 2024; Pang et al. 2023; Xu et al. 2024; Yi et al. 2023; Zhang et al. 2025].Most generate motion in pose-coordinate or learned latent spaces; DynaConTalk instead uses an explicit temporally aligned multiscale representation as the diffusion target.Source: §2.1 · p. 2
2.2 Content, Rhythm, and Dynamic Speech Conditioning
- Co-speech methods combine prosody, acoustic, self-supervised, text, and speaker cues; DynaConTalk dynamically regulates multiple conditions according to evolving motion state.Prior approaches include multimodal fusion and DisCo's separation of content and rhythm representations.Source: §2.2 · p. 2
2.3 Frequency and Multiscale Motion Representation
- Frequency-structured motion modeling includes Fourier and wavelet approaches, while DynaConTalk uses frame-aligned SWT coefficients to support generation and localized control.Unlike global Fourier representations or decimated wavelets, SWT retains coefficients at every frame and scale; matched-noise injection unifies history, keypose, and reference constraints.Source: §2.3 · pp. 2–3
3 Method
DynaConTalk generates holistic co-speech motion by diffusing aligned SWT coefficients, dynamically conditioning on speech and motion state, and applying matched-noise constraints for continuation and control.
3.1 Problem Formulation
- The formulation generates body-hand pose, facial expression, and root trajectory from frame-aligned audio features, transcript context, and speaker identity.Root translation is regressed from generated body pose because audio is largely uninformative for it and its three dimensions can be underweighted against the 330-dimensional body representation.Source: §3.1 · p. 3
3.2 Wavelet-Domain Diffusion
- Diffusion predicts clean motion in aligned SWT bands, separating coarse posture, gesture strokes, and fine details before inverse SWT reconstruction.The bands expose multiple motion scales in a shared generation space.Source: §3.1 · p. 3, §3.2 · p. 3
3.3 Structure-Aware Denoiser
- Body-hand denoising uses joint- and band-aware tokens with separate wavelet heads, while a lighter face stream predicts expression coefficients; a pose-driven BiGRU supplies root translation.The cA3 head models broad posture, cD3 and cD2 gesture strokes, and cD1 fast hand and facial details.Source: §3.3 · pp. 3–4, Figure 2 · p. 4
3.4 State-Aware Dynamic Gating and Consensus Conditioning
- HuBERT and speaker identity form a persistent base, while independent gates select rhythm, mel, and transcript cues using noisy motion and diffusion time.The residual gates can activate multiple cues or attenuate them while retaining the base.Source: §3.4 · pp. 4–5
- Attention pooling and learned stage routing adapt conditions to denoising depth, a frame-level rhythm path preserves timing, and signed consensus can preserve, suppress, or reverse proposed motion updates.The separate rhythm path bypasses condition pooling, which can dilute short rhythm events.Source: §3.4 · p. 5
3.5 Matched-Noise Constraint Injection
- Matched-noise injection re-noises clean history or reference targets to the current diffusion level, supporting window continuation, keypose repair, and reference-guided control.Keypose constraints are distributed across time, body parts, and wavelet bands; low-frequency pose is anchored while high-frequency details can be repaired by the denoising prior.Source: §3.5 · pp. 5–7
3.6 Training Objective
- The training objective predicts clean SWT samples and adds body position, velocity, boundary, foot-contact, and sparse-keypose supervision, with gate regularization for the face stream.Foot-contact and keypose losses belong to the editable body recipe; the speech-only recipes omit keypose supervision.Source: §3.6 · p. 7
4 Experiments
Experiments compare holistic generation on BEAT2 and SHOW, then test component contributions, wavelet representation effects, and long-form and controllable generation. The evaluation includes body and facial metrics, user ratings, temporal continuity, and editing behavior.
4.1 Experimental Setup
- The study uses BEAT2 and SHOW with official splits, and evaluates body motion using FGD, beat consistency, and diversity, plus facial MSE and LVD.Facial evaluation initializes history with the first eight expression frames and uses predicted expressions with ground-truth jaw pose and body shape.Source: §4.1 · p. 7
4.2 Overall Holistic Generation Quality
- DynaConTalk achieves the best FGD and facial reconstruction metrics among compared methods on both BEAT2 and SHOW.On BEAT2, the speech-only body row reports FGD 0.175, BC 0.536, diversity 7.85, facial MSE 3.55 × 10−8, and LVD 1.32 × 10−5.Source: §4.2 · p. 8, Table 1 · p. 8, Table 2 · p. 8
- Thirty participants rated DynaConTalk highest across four user-study criteria, while normalized foot sliding was less than half the best baseline.The ratings covered realism, semantic congruence, synchrony, and diversity; DynaConTalk also had the lowest absolute and 95th-percentile sliding.Source: §4.2 · p. 8, Table 3 · p. 8
4.3 Full-System Ablation Analysis
- Removing DGN causes the largest FGD degradation, followed by removing proposal–consensus; replacing DGN with cross-attention also underperforms the full model.Removing SWT has a smaller effect on FGD, with its fine-scale contribution examined separately.Source: §4.3 · p. 9, Table 5 · p. 9
- Temporal downsampling has the largest tested backbone effect, while spatial attention, band-specific heads, and window sampling improve generation quality.Three-level db6 gives the strongest complete wavelet result, although several other three-level families remain competitive.Source: §4.3 · p. 9, Table 6 · p. 9, Table 8 · p. 10
- Removing rhythm causes the largest beat-consistency drop with FGD nearly unchanged, whereas removing mel or HuBERT increases FGD by 0.047 or 0.088.Removing CLIP emotion or word-index cues improves FGD by 0.108 and 0.085, respectively.Source: §4.3 · p. 9, Table 7 · p. 10
4.4 Analysis of the SWT Motion Representation
- SWT bands capture progressively faster dynamics from slow cA3 posture to rapid cD1 motion; cD1 has 65–101× lower standard deviation than cA3 and concentrates 69–81% of energy in the highest-energy 10% of frames.With matched pose RMSE, temporal residual is 44.10 for Raw6D and 4.67 for SWT, an 89.4% reduction; SWT also yields W1 0.0887 versus 1.1217 for Raw6D in acceleration-kurtosis distributions.Source: §4.4.1 · p. 10, §4.4.2 · p. 11
4.4.2 Representation-Space Error Analysis.
- FGD responds about 20× less to 4–12 Hz than to 0.25–2 Hz perturbations; replacing predicted 3–15 Hz bands leaves 99.51% of FGD unchanged.Matched-noise constraints reduce continuation boundary position error by about one third, keypose edit spill by 42%, and reference-control style distance by about one third.Source: §4.4.2 · p. 11, §4.5 · p. 11, Table 10 · p. 11
5 Limitations
The section identifies limited locomotion and unevaluated generalization as scope boundaries, alongside analyses of motion-scale structure and acceleration dynamics.
5 Limitations
- Limited global locomotion in holistic co-speech datasets leaves root-trajectory generation weakly constrained by speech alone; broader-speaker and locomotion-heavy generalization remains unevaluated.The authors propose larger-scale training and stronger trajectory priors as future work.Source: §5 · p. 11
- 65–101× smaller cD1 standard deviation than cA3 coexists with 69–81% of cD1 energy in the highest-energy 10% of frames.This pattern holds across torso, arms, and hands.Source: Figure 8 · p. 12
- DynaConTalk closely follows the GT ECDF of log(1 + κacc), whereas Raw6D shifts toward higher acceleration kurtosis.The figure caption characterizes Raw6D's dynamics as more impulsive and heavy-tailed.Source: Figure 9 · p. 12
- Fig. 10 focuses on local smoothness around keypose editing.Source: Figure 10 · p. 12
6 Conclusion
DynaConTalk combines wavelet-domain diffusion and adaptive conditioning with matched-noise constraints to address motion-detail loss, static speech fusion, and window discontinuities. Controlled analyses link components to motion shape, activity calibration, smoother transitions, and localized control.
6 Conclusion
- DynaConTalk combines SWT-domain diffusion, dynamic state-aware gating, attention pooling and depth routing, signed proposal-consensus conditioning, and matched-noise constraints to address multiscale-detail loss, static fusion, and window discontinuities.Experiments support improved generation quality.Source: §6 · p. 12
- Fig. 11 presents qualitative examples of reference-guided control.Source: Figure 11 · p. 12
- Controlled analyses associate dynamic shape with SWT representation, activity calibration with DGN, and smoother window transitions and localized control with matched-noise constraints.Source: §6 · p. 12