PaperlayerView exact revision on arXiv

Source-linked AI summary

InterSing: Explicit Interaction Dynamics for 3D Duet Singing Animation and Beyond

Yihan Zhou, Zikai Huang, Yuyang Yu, Xuemiao Xu, Cheng Xu, Shengfeng He

arXiv:2609.04903v1cs.CV

TL;DR

Duet singing requires performers to balance individual expressiveness with sparse, rhythm-dependent interaction, which existing audio-driven and conversational models do not adequately represent. InterSing learns interpretable interaction logits with weak supervision and conditions an interaction-aware diffusion model on audio and interaction dynamics. The resulting framework supports controllable multi-mode generation with stronger coordination and musical alignment while preserving individual motion styles, and extends to multi-singer performances.

  • Problem

    Existing audio-driven animation and conversational interaction models do not adequately capture duet singing’s sparse, rhythm-dependent coordination between performers.

  • Method

    InterSing learns time-varying interaction logits through weakly supervised contrastive learning and conditions an interaction-aware diffusion model jointly on audio and interaction dynamics.

  • Results

    InterSing outperforms baselines across dyadic-generation metrics, including a 16.1% LVE reduction relative to UniLS, from 9.94 to 8.33.

  • Takeaways & Limitations

    The formulation enables controllable independent, coordinated, and transitional motion, preserves singers’ characteristic styles, and extends naturally to multi-singer performances.

  • Takeaways & Limitations

    The FLAME representation does not explicitly capture fine-grained eye-gaze dynamics, and large-scale multi-person chorus datasets are unavailable for comprehensive multi-singer evaluation.

Abstract

from arXiv · show

We present InterSing, a framework for generating realistic 3D head animations for duet singing performances. Unlike solo singing, duet performance requires each singer to balance individual expressiveness with intermittent interaction at musically salient moments, such as phrase boundaries, synchronized rhythms, and call-and-response passages. Because these interactions are sparse and rhythm-dependent, existing audio-driven animation methods and conversational interaction models do not adequately capture their structure. Our key insight is that duet coordination can be represented as a time-varying signal that reflects how strongly performers engage with one another throughout a song. Based on this observation, we introduce interaction logits, an interpretable latent representation that models the degree of cross-performer engagement at each time step. We learn these logits using weak supervision and use them to condition an interaction-aware diffusion model jointly driven by audio features and interaction dynamics. This formulation enables unified multi-mode generation, spanning independent motion, coordinated behavior, and smooth transitions between them. Experiments show that InterSing generates realistic and expressive singing head animations with stronger coordination and musical alignment than existing methods, while preserving each performer's characteristic motion style. We further demonstrate that the same formulation generalizes to multi-singer performances and provides intuitive control over when and how performers engage.

I. INTRODUCTION

InterSing treats duet-singing head animation as interaction-conditioned motion generation because singers must preserve individual expressiveness while coordinating intermittently at musically salient moments. It learns interpretable interaction logits and uses them to control unified generation across independent, coordinated, and transitional behaviors.

  • Motivation: Duet singing requires individual expressiveness alongside intermittent coordination at phrase boundaries, choruses, call-and-response passages, and synchronized rhythms.Coordination includes head orientation, facial affect, rhythmic emphasis, and synchronized motion.
  • Motivation: Existing dyadic methods rarely expose when and how strongly performers engage, limiting controllability and risking either independent or overly coupled motion.Production users may want to adjust interaction timing, frequency, or intensity without resynthesizing.
  • Core idea: Interaction logits provide a time-varying, interpretable control signal for coordination strength, enabling independent motion, coordinated duet behavior, and smooth transitions within one framework.The signal separates performer motion from interaction dynamics.
  • Interaction learning: Weakly supervised contrastive learning infers interaction logits from paired motion using geometric orientation anchors as sparse cues rather than ground-truth labels.The framework addresses subjective, difficult-to-annotate interaction and combines geometric priors with contrastive learning.
  • Generation framework: An interaction-aware diffusion framework jointly conditions on audio and interaction logits, with diffusion forcing supporting full duet synthesis, solo-performance refinement, and partner prediction.These modes share one generation framework and support controllable interaction dynamics.

II. RELATED WORK

Prior work progressed from rule-based and single-person audio-driven animation toward interactive multi-person facial motion, but singing and duet coordination remain challenging. InterSing addresses this gap with explicit, controllable modeling of cross-performer relationships while preserving individual singing characteristics.

  • Audio-driven animation: Rule-based and later deep-learning methods advanced audio-driven facial animation, but general speech models remain less effective for singing’s prosody and emotional expression.Singing-specific methods incorporate pitch, emotional transitions, lyrics, or audio to improve single-character animation.
  • Interactive animation: Conversational facial-animation methods model listener responses and contextual interaction, but their assumptions and required motion inputs limit applicability to real-time or singing scenarios.DualTalk requires precomputed interlocutor motion, while UniLS models interactive animation from dual-channel speech audio.
  • Duet singing: PaChorus introduced joint duet-singing head-motion synthesis, but its implicit interaction modeling can produce limited coordination and weak temporal alignment.InterSing instead formulates the task around explicit cross-performer relationships.
  • Duet singing: InterSing’s explicit interaction representation enables fine-grained modulation of inter-singer dynamics while preserving each performer’s individual singing characteristics.The representation is designed to be interpretable and controllable.

III. METHOD

InterSing uses weak geometric supervision and contrastive objectives to learn temporally coherent interaction representations, then conditions diffusion-based motion generation on these signals. The two-stage design separates interaction dynamics from individual motion while supporting interpretable duet coordination.

  • III. METHOD: InterSing uses FLAME-based motion representations in a two-stage framework for weakly supervised interaction learning followed by interaction-aware diffusion generation.The motion representation includes facial expression, neck-pose, and jaw-pose parameters.
  • III. METHOD: Relative neck orientations provide sparse pseudo-labels, with mutual rotation angles above a threshold treated as interaction and the remainder as non-interaction.These anchors guide learning without being treated as strict ground-truth labels.
  • III. METHOD: A Transformer motion encoder masks 40% of neck features to discourage geometric shortcuts and promote learning from facial, temporal, and synchronized-motion cues.The encoder maps both singers’ motion sequences into a joint latent space.
  • III. METHOD: Stage II injects learned interaction scores into a Diffusion Forcing model with cascaded cross-attention to jointly denoise both singers’ motions under variable noise horizons.Variable noise horizons support flexible generation modes conditioned on interaction dynamics.
  • III. METHOD: Dual projection heads produce performer embeddings that pseudo-labels partition into interaction and non-interaction sets for contrastive optimization.The learned embeddings are intended to capture higher-level interaction semantics beyond direct spatial alignment.
  • III. METHOD: The intra-temporal alignment loss pulls same-frame interaction embeddings together while separating temporally distant pairs beyond a distance threshold.This objective promotes temporal coherence and discriminability in the interaction representation.
  • III. METHOD: The inter-discriminative loss pulls simultaneous interaction pairs together and pushes non-interaction pairs apart to sharpen state separation.Together, the two losses transform sparse geometric anchors into a temporally coherent and interpretable representation.

B. Stage II: Interaction-aware Diffusion Generation

InterSing uses a Transformer-based conditional diffusion model to generate coordinated duet head motions from audio and interaction signals. Token-wise noise and interaction-aware objectives support flexible temporal coupling while enforcing motion fidelity and coordination.

  • Interaction conditioning: Interaction intensity is represented continuously per frame and can be predicted from speaker-aware audio when motion-derived supervision is unavailable at inference.An auxiliary audio-to-score decoder estimates interaction dynamics from rhythmic and phrasing cues.
  • Diffusion Forcing: Diffusion Forcing independently samples noise across frames and singers, representing both tightly synchronized and loosely coupled motion segments.This token-wise stochasticity supports heterogeneous corruption across temporal and motion dimensions.
  • Interaction-aware diffusion: The model jointly denoises both singers’ motions using audio features, interaction signals, diffusion timesteps, and cross-stream attention.Self-attention preserves temporal coherence, while audio and interaction cross-attention inject motion dynamics and dyadic coupling.
  • Flexible inference: Different noise schedules enable dyadic generation, partner generation, and score-guided generation within the same framework.These modes arise from the flexible inference design rather than separate generation formulations.
  • Training objectives: The training objective combines reconstruction, velocity, acceleration, alignment, and interaction-consistency losses to enforce accurate, smooth, and coordinated motion.Interaction consistency matches intensity extracted from generated motion with the desired interaction score.

C. Inference

InterSing uses one pretrained diffusion model across multiple inference modes by changing noise initialization and conditioning. This supports synchronized duet synthesis, partner generation, and interaction-aware refinement without task-specific retraining.

  • Unified inference: A single architecture supports multiple inference modes by adjusting noise initialization and conditioning strategy.The unified formulation is designed to preserve temporally coherent and interaction-consistent motion generation.
  • Interaction-aware refinement: Low-interaction regions remain largely unchanged during refinement, while the diffusion model improves interaction awareness and preserves temporal consistency.This describes interaction-aware refinement of an existing performance.
  • Training-free adaptation: The framework adapts to different input conditions without task-specific fine-tuning or architectural modification.The same pretrained model is used across all inference modes.

1) Dataset:

The authors adapt ChorusHead and supplement it with real duet videos to construct an interaction-aware dataset of reliable dyadic singing sequences. The resulting corpus spans diverse performers, songs, languages, and musical styles under a song-disjoint split.

  • Dataset construction: ChorusHead is adapted from collective chorus representation to pairwise interaction modeling through targeted preprocessing and interaction-aware supervision.The adaptation extracts reliable dyadic singing sequences from the original dataset.
  • Data curation: The preprocessing removes pseudo-duets, non-interactive back-to-back performances, and unstable side-profile reconstructions.Real duet videos are added to increase scale and coordination-pattern diversity.
  • Reconstruction and audio processing: Full-frame MTCNN detection produces more coherent 3D head motion sequences, while audio separation extracts individual singer tracks and speaker activity scores.These processing choices support per-performer motion and audio conditioning.
  • Dataset composition: The adapted dataset contains 9 hours, 414 clips, 149 songs, 972,394 frames, 61 singers, and 39 groups.It covers R&B, pop, rock, and other genres, with English, Chinese, and Japanese samples.
  • Implementation details: Training uses two AdamW stages on two RTX 4090 GPUs, and inference exceeds 250 FPS on one RTX 4090 across all three modes.The reported threshold and loss-weight settings define the implementation configuration.

3) Evaluation Metrics:

The evaluation covers facial accuracy, head pose, rhythm alignment, temporal quality, and interaction consistency using established metrics and baselines. Quantitative and qualitative comparisons assess both dyadic and partner generation.

  • Evaluation metrics: Evaluation measures facial accuracy, head pose accuracy, rhythm alignment, temporal coherence and quality, and interaction consistency.Facial accuracy uses LVE, FDD, FVE, and Freeze Rate for lip, facial-dynamics, geometry, and freezing behavior.
  • Baselines: Baselines include PaChorus and conversational interaction methods DualTalk, DIM, and UniLS, all retrained on the same dataset.The comparison covers methods conditioning on counterpart motion or both participants’ audio.
  • Quantitative results: 16.1% lower LVE than UniLS, 8.33 versus 9.94, is reported for dyadic generation.The passage also reports 29.2% and 25.9% relative reductions in FID and P-FID.
  • Qualitative results: Qualitative comparisons report better ground-truth matching in lip synchronization, facial-expression dynamics, and mutual interaction for dyadic generation.The evaluation also examines partner responses, expression continuity, and responsive head dynamics.

2) Qualitative Results:

InterSing produces more synchronized, expressive, and naturally coordinated duet animations than existing methods, while allowing interaction strength to vary under identical audio. Ablations further show that explicit interaction modeling and score conditioning support stable, long-range coordination.

  • InterSing generates more natural interaction patterns, synchronized facial movements, and coherent responses than existing methods in qualitative comparisons.
  • Under identical audio, generated facial motions adapt to no-interaction, low-interaction, and high-interaction score conditions while maintaining audio-driven dynamics.
  • Ablation Study: Removing interaction cross-attention produces the worst P-FID, indicating that naive feature concatenation fails to model explicit cross-person dependencies.
  • Ablation Study: Removing score injection preserves competitive FID/P-FID but increases Freeze Rate and FDD, exposing weaker long-range coordination.
  • Ablation Study: Ground-truth interaction scores yield the best FID and P-FID, supporting the role of precise interaction priors in diffusion guidance.

V. EXTENSION TO MULTI-PERSON SCENARIOS

InterSing extends duet interaction modeling to multi-person singing by composing sequential pairwise generations without additional training. On a collected three-singer subset, it outperforms PaChorus across all reported metrics, while the paper notes important representation and dataset limitations.

  • V. EXTENSION TO MULTI-PERSON SCENARIOS: Sequential pairwise generation reuses the pretrained dyadic framework to provide a training-free extension to multi-person singing.Interactions are restricted to immediate neighbors, decomposing global coordination into pairwise steps.
  • V. EXTENSION TO MULTI-PERSON SCENARIOS: InterSing explicitly composes pairwise interaction scores, whereas PaChorus uses a single partner representation for all nontarget singers.
  • V. EXTENSION TO MULTI-PERSON SCENARIOS: InterSing consistently outperforms PaChorus on all metrics in the collected three-singer subset.
  • VI. LIMITATION: Fine-grained eye-gaze dynamics are not explicitly captured because of limitations inherent to the FLAME representation.
  • VI. LIMITATION: The lack of large-scale multi-person chorus datasets prevents comprehensive quantitative evaluation in multi-singer scenarios.
  • VII. CONCLUSION: The conclusion reports consistent improvements in lip synchronization, facial expressiveness, and perceived interaction realism, with ablations confirming explicit interaction modeling and contrastive learning.
Loading 2609.04903v1…