Source-linked AI summary
CoDance: An Unbind-Rebind Paradigm for Robust Multi-Subject Animation
Shuai Tan, Biao Gong, Ke Ma, Yutong Feng, Qiyuan Zhang, Yan Wang, Yujun Shen, Hengshuang Zhao
TL;DR
Existing character animation methods struggle with varying subject counts, types, and spatial misalignment because rigid pose–reference binding impedes motion transfer and subject association. CoDance addresses this with Unbind-Rebind: perturbed pose and feature representations learn location-agnostic motion, while text prompts and subject masks restore semantic and spatial control. Experiments on Follow-Your-Pose-V2 and CoDanceBench report state-of-the-art performance and strong generalization across diverse subjects and layouts.
Problem
Existing methods struggle to animate arbitrary subject counts and types when the driving pose is spatially misaligned with the reference image.
Method
CoDance uses Unbind to perturb poses and latent features during training, then Rebinds motion with semantic text guidance and spatial subject masks.
Results
CoDance achieves state-of-the-art performance on CoDanceBench and Follow-Your-Pose-V2, with strong generalization across diverse subjects and spatial layouts.
Takeaways & Limitations
The Unbind-Rebind paradigm supports flexible multi-subject animation under varied subject counts, types, poses, and spatial configurations.
Takeaways & Limitations
Existing two-person methods remain constrained by scalability beyond two subjects, strict initial spatial alignment, and poor generalization to non-human subjects.
Abstract
from arXiv · showhide
Character image animation is gaining significant importance across various domains, driven by the demand for robust and flexible multi-subject rendering. While existing methods excel in single-person animation, they struggle to handle arbitrary subject counts, diverse character types, and spatial misalignment between the reference image and the driving poses. We attribute these limitations to an overly rigid spatial binding that forces strict pixel-wise alignment between the pose and reference, and an inability to consistently rebind motion to intended subjects. To address these challenges, we propose CoDance, a novel Unbind-Rebind framework that enables the animation of arbitrary subject counts, types, and spatial configurations conditioned on a single, potentially misaligned pose sequence. Specifically, the Unbind module employs a novel pose shift encoder to break the rigid spatial binding between the pose and the reference by introducing stochastic perturbations to both poses and their latent features, thereby compelling the model to learn a location-agnostic motion representation. To ensure precise control and subject association, we then devise a Rebind module, leveraging semantic guidance from text prompts and spatial guidance from subject masks to direct the learned motion to intended characters. Furthermore, to facilitate comprehensive evaluation, we introduce a new multi-subject CoDanceBench. Extensive experiments on CoDanceBench and existing datasets show that CoDance achieves SOTA performance, exhibiting remarkable generalization across diverse subjects and spatial layouts. The code and weights will be open-sourced.
1. Introduction
CoDance targets robust character animation when subject count, type, and spatial layout vary, addressing failures caused by rigid pose–reference binding and unreliable subject association. Its Unbind-Rebind design separates motion understanding from subject control using perturbed pose representations, semantic prompts, and spatial masks.
- Existing animation methods struggle with arbitrary subject counts and spatial layouts, often producing spurious or entangled subjects in multi-subject scenes.
- Two-person methods remain limited by rigid pose-conditioned binding, sensitivity to reference–pose misalignment, and restricted scalability beyond two subjects.
- Prior methods also generalize poorly from real humans to non-human or anthropomorphic characters, constraining practical applicability.
- Unbind: CoDance unbinds motion from strict spatial correspondence through stochastic scale and shift perturbations applied to poses and latent features.This encourages subject-agnostic motion semantics and temporal coherence while reducing reliance on pixel alignment.
- Rebind: Its Rebind module restores control using text prompts for subject identity and count, plus subject masks for precise spatial guidance.Mixed-data training with text-to-video data strengthens textual understanding and semantic binding.
- CoDance achieves arbitrary subject types, counts, spatial positions, and poses from a single pose sequence that need not align spatially with the reference image.Experiments on CoDanceBench and existing benchmarks report state-of-the-art performance across metrics and strong generalization.
2. Related Work
Video generation methods have evolved from temporal adaptations of image models toward scalable diffusion-transformer backbones. Character animation has similarly moved from GAN pipelines to diffusion-based, controllable motion transfer, but multi-subject methods remain largely limited by rigid pose–image binding.
- Video synthesis requires smooth trajectories and temporal consistency in addition to per-frame visual quality.
- One major approach augments text-to-image backbones with lightweight temporal attention or 3D-convolution adapters.
- Diffusion Transformers replace U-Nets with Transformer backbones that scale more gracefully to long sequences and complex dynamics.CoDance adopts Wan 2.1 as its backbone for its reported motion fidelity.
- Character animation progressed from GAN-based pipelines with distortion, identity drift, and flicker toward diffusion models with stronger priors and controllable conditioning.
- Existing multi-character approaches use optical flow, depth ordering, masks, prompt decomposition, and multi-branch control, but struggle beyond two subjects because of rigid pose–image binding.
3. Methodology
CoDance combines a pose-shift-based Unbind module with semantic and spatial Rebind guidance in a DiT-based animation pipeline. Training disrupts pose–reference alignment to learn motion semantics, then restores subject control with prompts and masks.
- Framework: The pipeline takes a reference image, driving pose sequence, text prompt, and subject mask to generate an animation video.
- Framework: A VAE encodes the reference image, while the Pose Shift Encoder produces pose features that are combined with noisy latent tokens for the DiT backbone.
- Rebind: Rebind injects semantic text features through cross-attention and spatial mask features element-wise into the noisy latent.
- Scope: CoDance explicitly supports arbitrarily many diverse subjects, misaligned inputs, non-human characters, and multi-subject scenes while maintaining reference identity consistency.
- Unbind: Unbind breaks rigid pose–reference alignment so the model learns semantic motion rather than hallucinating pose-aligned subjects under mismatched inputs.
- Unbind: Feature Unbind randomly translates and duplicates pose regions in feature space, forcing the diffusion model to adapt to varied pose configurations.
- Rebind: Rebind uses text prompts to specify the identity and number of animated subjects and masks to define their animation regions.
- Implementation: Unbind modules and mixed-data training operate only during training and are bypassed during inference, adding no inference computational overhead.
4. Experiments
Experiments evaluate CoDance against existing methods on single- and multi-subject benchmarks, including the proposed CoDanceBench, with quantitative, qualitative, user-study, and ablation analyses. Results show stronger multi-subject identity preservation, motion fidelity, perceptual quality, and temporal consistency, while isolating the roles of Unbind and Rebind.
- Evaluation setup: CoDance is evaluated on Follow-Your-Pose-V2 and the newly introduced CoDanceBench, which contains 20 multi-subject dance videos.The evaluation also includes selected TikTok and Fashion videos for single-person animation.
- Quantitative results: The quantitative comparison covers perceptual similarity, identity consistency, and motion fidelity against established state-of-the-art animation methods.Reported metrics include LPIPS, PSNR/SSIM, FID-FVD, and FVD.
- Quantitative results: CoDance significantly outperforms competing methods across LPIPS, PSNR/SSIM, FID-FVD, and FVD in multi-agent evaluations.The comparison uses full driving-video pose skeletons, placing primarily single-person architectures in a multi-subject setting.
- Qualitative comparison: Conventional methods often produce identity confusion, visual artifacts, and motion distortion, whereas Unbind-Rebind preserves distinct identities and coherent motion for each character.The reported robustness is achieved despite training CoDance and the baselines under the same single-person data constraint.
- Generalization: Across one-to-five-subject cases, rigidly bound methods overlook subject identity and count, while Animate-X treats multi-subject scenes as a monolithic entity without rebinding motion to individuals.CoDance is contrasted with these failure modes in settings involving spatial misalignment and a single driving skeleton.
- User study: User studies report the highest CoDance preference rates for video quality, identity preservation, and temporal consistency.Participants compared side-by-side videos generated by different methods.
- Ablation study: The progressive ablation removes both modules, then adds Unbind and spatial Rebind to isolate their contributions to animation quality.The baseline follows a rigid alignment paradigm, after which Unbind breaks reference-pose alignment and spatial Rebind adds mask conditioning.
5. Conclusion
CoDance addresses identity degradation and motion misassignment in multi-subject animation by unbinding motion from rigid spatial context and rebinding it with semantic and spatial guidance. Extensive experiments on Follow-Your-Pose-V2 and CoDanceBench report improvements over state-of-the-art methods.
- CoDance targets robust animation across arbitrary subject counts, types, and spatial layouts.
- Its Unbind-Rebind paradigm first decouples motion from rigid spatial context, then directs it to the correct subjects using semantic and spatial guidance.
- Extensive experiments on Follow-Your-Pose-V2 and CoDanceBench demonstrate improvements over state-of-the-art methods.