PaperlayerView exact revision on arXiv

Source-linked AI summary

InteractGesture: Progressive Chunk Guidance for Continuous Streaming Co-Speech Gesture Control

Ekkasit Pinyoanuntapong, Ajinkya Deogade, Paul Streli, Wenjing Zhang, Joanna Materzynska, Pu Wang, Vittorio Ferrari, Jie Shen

arXiv:2608.25734v1cs.CV

TL;DR

Co-speech gesture generators lack fine-grained spatial control of individual joints, especially when continuous streaming freezes earlier chunks before future constraints are known. InteractGesture guides latent estimates through a differentiable decoder, while Progressive Chunk Guidance keeps overlapping chunks editable during streaming. On BEAT2, the method improves spatial control while preserving gesture quality and supports several joint-level control tasks.

  • Problem

    Existing co-speech gesture generators lack fine-grained spatial controllability of individual joints, and sequential streaming inference freezes earlier chunks before future constraints can adjust them.

  • Method

    InteractGesture applies inference-time spatial losses through a differentiable decoder to update target latent estimates, while Progressive Chunk Guidance maintains editable active chunks across streaming boundaries.

  • Results

    Progressive Chunk Guidance achieves 6.34 cm average error while improving gesture naturalness with FGD 0.431 and Diversity 11.760 over Synchronous guidance.

  • Takeaways & Limitations

    The framework supports sparse joint positioning, dense trajectory tracking, directional pointing, and absolute or root-relative spatial targets.

Abstract

from arXiv · show

Co-speech gesture generation has made significant progress toward realistic full-body motion from speaker audio, yet existing models lack fine-grained spatial controllability of individual joints. To address this, we introduce \emph{InteractGesture}, a model-agnostic, inference-time method for spatially controllable gesture generation. \emph{InteractGesture} guides target latent estimates of a diffusion sampler through a differentiable RVQ-VAE decoder, backpropagating spatial control gradients to adjust motion latents during sampling. A primary challenge in streaming co-speech generation is chunk-wise dependency: standard sequential inference freezes prior chunks, preventing spatial constraints in future chunks from adjusting preceding trajectories and causing boundary inconsistencies. To overcome this limitation, we propose \emph{Progressive Chunk Guidance}, a chunk-window strategy that maintains an active set of editable chunk latents with staggered delays, enabling spatial constraints to propagate gradients backward across chunk boundaries during streaming generation. Experiments on the BEAT2 dataset show that \emph{InteractGesture} improves multi-joint spatial control while preserving overall gesture quality. Furthermore, our approach supports diverse applications, including sparse joint positioning, dense joint trajectory control, and directional pointing. Our project page is available at https://exitudio.github.io/interactgesture-page .

1 Introduction

InteractGesture adds inference-time fine-grained spatial control to pretrained co-speech gesture generators, addressing the difficulty of maintaining control across streaming chunk boundaries. Progressive Chunk Guidance keeps active chunks editable so future constraints can adjust preceding motion while preserving streaming execution.

  • Existing co-speech gesture generators produce plausible motion but lack fine-grained spatial controllability of individual body parts.
  • InteractGesture guides target latent estimates through a differentiable decoder, applying masked spatial losses while leaving pretrained model parameters unchanged.
  • Standard sequential chunk inference freezes chunk k before generating chunk k + 1, preventing later constraints from adjusting earlier trajectories.
  • Synchronous guidance propagates controls across chunks but requires full-sequence audio offline, limiting its use for continuous streaming.
  • Progressive Chunk Guidance maintains editable active chunks, introduces new chunks with staggered delays, and refreshes context from guided estimates to propagate control across boundaries.
  • The framework supports absolute scene-space and root-relative targets, sparse joint positioning, dense trajectory control, and directional pointing.

2 Related Work

Prior work broadens speech-driven gesture generation and offline motion control, but direct fine-grained 3D control of individual joints during continuous streaming remains insufficiently addressed. InteractGesture adapts inference-time latent guidance and differentiable decoding to this setting.

  • Co-speech gesture research has improved alignment and expressiveness through stronger audio-motion representations, semantic cues, contrastive objectives, and conversational context.
  • Recent methods primarily control identity, style, emotion, semantic intent, or interaction context rather than direct 3D spatial targets for individual joints.
  • Text-to-motion methods support trajectories, inbetweening, body-part masks, programmable constraints, and keyframes, but typically assume static prompts, fixed lengths, and offline generation.
  • Controllable co-speech methods study pointing, trajectory control, multimodal examples, and object-aware interaction without providing fine-grained multi-joint 3D control during continuous streaming.
  • InteractGesture optimizes target latent estimates through a differentiable decoder, adapting sampling-time motion guidance to co-speech latent generation.

3 Method

InteractGesture guides a pretrained streaming gesture generator by optimizing target latent estimates through a differentiable decoding path, while representing controls as masked joint-position constraints. Progressive Chunk Guidance keeps multiple chunk latents editable with staggered activation, allowing future constraints to adjust earlier trajectories and improve continuity within streaming generation.

  • Latent Guidance: InteractGesture backpropagates spatial-control gradients through RVQ-VAE and SMPL-X decoding into target latent estimates while keeping model parameters fixed.The differentiable path maps latent estimates to 3D joints, evaluates masked spatial loss, and updates the target estimate rather than the generator.
  • Streaming Setup: Each streaming pose chunk contains 128 frames, uses 32-frame latent windows, and combines overlapping windows with stride R = P − S.For noninitial chunks, constraints overlapping historical context are masked out to avoid double-counting while preserving sequence-level control.
  • Control Representation: The control representation stores target joint positions in H and active joint-frame constraints in binary mask M, supporting sparse keyframes and dense trajectories.M entries equal to 1 contribute to the control loss, while entries equal to 0 remain unconstrained.
  • Optimization Schedule: Spatial optimization uses Adam with learning-rate normalization by active constraint count and increases latent optimization effort toward later denoising steps.Later steps receive more guidance because decoded joints are more reliable near the clean target manifold.
  • Progressive Chunk Guidance: Progressive Chunk Guidance maintains an active window of concurrently denoised, editable chunk latents and introduces new chunks with staggered sampler-step delays.New chunks condition on evolving guided estimates from preceding chunks, enabling backward gradient propagation across active chunk boundaries.
  • Control Modes: The method supports joint-position, dense-trajectory, and pointing-direction control during continuous streaming generation.These modes range from sparse wrist targets to temporally dense hand or elbow paths and directional pointing constraints.

4 Experiments

The BEAT2 evaluation measures spatial accuracy and motion quality across chunk-guidance schedules and control baselines. Progressive Chunk Guidance supports continuous streaming while balancing accurate target control with natural co-speech motion.

  • Evaluation Setup: Evaluation uses GestureLSM on BEAT2 with frozen generator and RVQ-VAE decoders, 128-frame chunks, and 30 FPS synthesis.The benchmark covers 15 settings formed by three keyframe densities and five controlled-joint configurations.
  • Evaluation Metrics: Metrics assess motion quality through FGD, Beat Consistency, and Diversity, alongside average control, location, and trajectory errors.Location and trajectory errors count targets exceeding 10 cm, while average control error is measured in centimeters.
  • Quantitative Results: 46.16 cm average error and 97.6% location failure rate show that Chunk-Wise ControlNet poorly reaches keyframe targets despite FGD 0.491.The comparison includes post-hoc IK, which records FGD 1.043 and 19.97 cm average control error.
  • Quantitative Results: 4.67 cm average error makes Synchronous Chunk Guidance the strongest offline schedule, while Sequential Chunk Guidance reaches 11.70 cm under frozen prior chunks.Sequential guidance improves spatial accuracy through test-time latent optimization but cannot revise earlier chunks for future constraints.
  • Quantitative Results: 6.34 cm average error lets Progressive Chunk Guidance enable continuous streaming, while FGD 0.431 and Diversity 11.760 improve gesture naturalness over Synchronous guidance.Its active overlapping chunks allow later controls to influence preceding motion across boundaries.
  • Qualitative Results: Figure 4 compares identical keyframe controls visually, showing that IK can place joints near targets while producing less natural surrounding motion.The qualitative comparison emphasizes whether wrists and elbows satisfy requested positions while retaining the learned speech-conditioned gesture prior.

5 Ablation Study

The ablation study on BEAT2 evaluates guidance components, guidance scales, and execution schedules for spatial control. Full Guidance provides the best component-level accuracy, while Progressive Chunk Guidance preserves comparable control and generation quality under streaming constraints.

  • Guidance Components: Table 2 compares guidance configurations at scale 0.5 under Delay 0 synchronous and Delay 1 progressive execution.The configurations are Uncontrolled Baseline, Sampling Guidance only, Post-Refinement only, and Full Guidance.
  • Guidance Components: 59.308 cm and 64.984 cm average errors without guidance show that spatial control fails under both execution schedules.
  • Guidance Components: 4.669 cm and 6.335 cm average errors make Full Guidance the most accurate configuration for Delay 0 and Delay 1, respectively.Sampling Guidance supplies coarse alignment during denoising, while Post-Refinement fine-tunes final target latent estimates.
  • Guidance Scale Sensitivity: 10.46 cm at scale 0.1 decreases to 4.07 cm at scale 1.0 in the offline Synchronous setting.Increasing guidance scale monotonically improves spatial precision in this setting.
  • Guidance Scale Sensitivity: Scale 0.5 is optimal for Progressive Chunk Guidance, combining 6.34 cm spatial error with FGD 0.431, while scale 1.0 reaches 7.31 cm error.
  • Execution Schedules: Progressive Chunk Guidance maintains streaming-compatible control with 6.335 cm average error, Traj10 0.267, Loc10 0.161, and FGD 0.431.Compared with Synchronous Chunk Guidance, it avoids requiring the full speech sequence in advance while preserving most control benefits.

6 Conclusion

InteractGesture enables model-agnostic inference-time spatial control for co-speech gesture generation, while Progressive Chunk Guidance keeps future chunks able to influence preceding motion during streaming. BEAT2 experiments show reduced joint errors while preserving speech-conditioned gesture naturalness across multiple control applications.

  • InteractGesture guides pretrained co-speech samplers through differentiable RVQ-VAE decoding and latent optimization for spatial control.
  • Progressive Chunk Guidance keeps overlapping chunk latents editable with staggered delays, allowing later spatial constraints to influence preceding motion without the full speech sequence.
  • The method supports sparse joint control, dense joint trajectory control, and pointing-direction control while reducing joint errors and preserving gesture naturalness on BEAT2.
Loading 2608.25734v1…