PaperlayerView exact revision on arXiv

Source-linked AI summary

SCAIL-2: Unifying Controlled Character Animation with End-to-End In-Context Conditioning

Wenhao Yan, Fengjia Guo, Zhuoyi Yang, Jie Tang

arXiv:2606.10804v3cs.CV

TL;DR

Character animation methods often depend on lossy intermediate motion representations, limiting handling of complex, multi-character, and non-human inputs. SCAIL-2 directly conditions on visual driving context, trains on synthesized motion pairs, and achieves state-of-the-art performance across diverse animation tasks.

  • Problem

    Existing character animation methods rely on ambiguous or limited intermediate representations, which struggle with complex motion, multiple characters, detailed movements, and out-of-distribution driving.

  • Method

    SCAIL-2 directly conditions on visual driving context, synthesizes MotionPair-60K for end-to-end training, and unifies subtasks with task-specific conditions.

  • Results

    SCAIL-2 achieves state-of-the-art performance across diverse animation tasks and input conditions, including complex motion, multi-character interactions, and non-human inputs.

  • Takeaways & Limitations

    The unified end-to-end formulation generalizes across zero-shot inputs and supports multiple character animation subtasks within one interface.

  • Takeaways & Limitations

    The framework depends on large-scale, high-quality paired training data, while reliable positive samples for fine-grained regions remain difficult to obtain.

Abstract

from arXiv · show

Controlled character animation aims to transfer motion from a driving sequence to a reference character. Prior works heavily rely on intermediate representations, such as pose skeletons for motion and masked backgrounds for environment, inevitably resulting in information loss. In this work, we present SCAIL-2, a framework that adopts an end-to-end driving paradigm by directly concatenating latent visual information to the model's input sequence. We enable end-to-end training through a data synthesis pipeline that produces MotionPair-60K, a curated dataset for several character animation subtasks. We unify the subtasks using decoupled conditions to accommodate different driving patterns, facilitated by In-Context Mask Conditioning and Mode-Specific RoPE, which provide soft guidance beyond textual instructions and visual information. To address synthetic discrepancy in detailed regions, we propose Bias-Aware DPO to construct preference items to mitigate the errors. Extensive experiments demonstrate that our method achieves state-of-the-art performance across various character animation tasks. Code, model weights, and a large subset of the dataset are available at: https://teal024.github.io/SCAIL-2/.

1 INTRODUCTION

SCAIL-2 proposes end-to-end visual conditioning for controlled character animation, avoiding lossy intermediate representations while unifying diverse animation tasks. It introduces MotionPair-60K, specialized conditioning mechanisms, Bias-Aware DPO, and an open-source pose-free model that achieves state-of-the-art performance across animation tasks.

  • Motivation: Existing character-animation methods rely on pose maps, bottleneck encoders, or test-time motion embeddings, which lose information and remain ambiguous in complex scenarios.Skeleton maps can be inaccurate, while bottleneck encoders lack detail according to the introduction.
  • End-to-End Paradigm: SCAIL-2 defines end-to-end animation as directly providing visual driving context instead of relying on information-losing intermediate representations.Training requires paired sequences of different characters performing the same motion, motivating a synthesis pipeline for motion-aligned videos.
  • Data Construction: The motion-pair synthesis pipeline produces MotionPair-60K, a heterogeneous dataset designed to support end-to-end training.The pipeline addresses scarce paired data for characters performing the same motion within or across environments.
  • Unified Conditioning: The framework unifies animation subtasks through task-specific conditions, In-Context Mask Conditioning, and Mode-Specific RoPE under reverse driving training.Environment Switch supports vanilla animation and replacement, while Binding Slots encode motion–character binding for single- and multi-character settings.
  • Results and Contributions: SCAIL-2 performs strongly on complex motion, multi-character interactions, and non-human inputs, while improving cross-identity motion transfer and environment integration.The work also introduces Bias-Aware DPO for detailed motion capture and releases an open-source pose-free animation model with state-of-the-art performance across animation tasks.

2 RELATED WORKS

Related work covers two character-animation settings: Animation Mode animates a character in its original background, whereas Replacement Mode places a reference character in the driving video's environment. Existing approaches commonly rely on pose-driven representations, while MoCha explores end-to-end replacement using Unreal Engine 5 data.

  • Character Image Animation: Animation Mode animates a character within its original background, following the terminology of Wan-Animate.
  • Character Image Animation: Existing Animation Mode methods typically extract 2D skeletal motion sequences from driving videos and inject them into video generation.SCAIL instead introduces an identity-agnostic 3D skeleton representation.
  • Character Replacement: Replacement Mode animates a reference character within the environment of the driving video, with prior methods combining pose-driven animation and background inpainting.
  • Character Replacement: MoCha trains an end-to-end character-replacement model using data rendered in Unreal Engine 5.

3 METHOD

SCAIL-2 formulates character animation as end-to-end conditional flow matching, directly encoding driving videos in shared VAE latent space. It unifies subtasks through binding and environment conditions, In-Context Mask Conditioning, Mode-Specific Shifted RoPE, synthetic training data, and Bias-Aware DPO.

  • General Task Formulation: The flow-matching backbone encodes input video x as z0 = E(x), samples z1 ∼ N(0, I), and trains a conditional velocity model on interpolated latents.The interpolation is zt = (1 − t)z0 + tz1 with velocity v = z1 − z0, where t ∼ U(0, 1) and c denotes auxiliary conditions.
  • General Task Formulation: Unlike pose-driven methods, the end-to-end solution directly encodes the driving video as zdriv = E(y), bypassing explicit pose estimation in shared VAE latent space.The condition also includes a text prompt, reference image, and motion signal derived from the driving video.
  • Subtask Formulation: SCAIL-2 unifies subtasks with a binding map π : Cy → CI and environment source E, optimizing motion binding, environment weaving, and universal transfer.These objectives route motions to bound targets, compose characters with either reference or driving environments, and disentangle pose from identity without identity leakage.
  • Data Synthesis and Preference Optimization: The synthetic pipeline constructs MotionPair-60K by reverse-driving synthesized videos, while Bias-Aware DPO targets hand-region errors from pose-driven synthetic data.The pipeline uses pose transfer or one-by-one replacement, and focuses preference correction on incorrectly articulated finger joints.
  • In-Context Mask Conditioning: In-Context Mask Conditioning adds an Environment Switch and K Binding Slots to distinguish subtasks and provide soft guidance while preserving complete visual information.The Environment Switch selects the reference or video environment, while Binding Slots support character-specific motion routing.
  • Mode-Specific Shifted RoPE: Mode-Specific Shifted RoPE models Animation Mode’s regenerated starting frame versus Replacement Mode’s preserved background and character-only regeneration.The design applies different temporal and spatial coordinate ranges to the denoising latent and reference for the two modes.

4 EXPERIMENTS

SCAIL-2 is evaluated on Studio-Bench and X-Dance across cross-identity, pose-driven, and replacement settings, with user studies used where cross-identity targets lack ground truth. Experiments show strong motion transfer, identity preservation, zero-shot generalization, and benefits from visual driving, Binding Slots, unified data, and Bias-Aware DPO.

  • Evaluation Protocol: Evaluation uses Studio-Bench and X-Dance, including adapted real-world cross-identity cases assessed through blind Good/Same/Bad user studies.Cross-identity animation lacks ground-truth targets, making user studies the most reliable evaluation.
  • Cross-Identity Results: SCAIL-2 outperforms leading open-source methods in single-character human evaluations, remains close to Kling 3.0, and shows clear zero-shot advantages in multi-character animation.X-Dance results also show competitive video quality, while replacement mode is preferred over inpainting-based alternatives.
  • Pose-driven Results: Conditioning on SAM3D-Body human meshes yields competitive low-level metrics against pose-driven baselines, despite limited pose-pair training and the model not being designed as a pose-driven generator.Compared methods include Wan-Animate, SCAIL, UniAnimate-DiT, VACE, and HyperMotion.
  • Qualitative Results: Qualitative results preserve motion, identity, fine-grained human–object interactions, character generalization, and natural environmental integration under cross-identity and replacement inputs.The model avoids failures including lost instruments and dark inpainting artifacts, while maintaining body-shape consistency and shoe reflections.
  • Zero-Shot Abilities: Zero-shot experiments generalize to cartoon animals, egocentric inputs, and object motion, including faithfully reversing and resuming a chair’s rotation despite human-only training.Direct visual motion transfer leverages pretrained priors beyond human-centric intermediate representations.
  • Ablations: Ablations show that visual driving improves motion fidelity, Binding Slots preserve identity, replacement data supports multi-character animation, and Bias-Aware DPO improves hand details and overall motion consistency.Removing Binding Slots reduces Appearance Consistency; DPO globally updates the policy while up-weighting salient hand-region errors and improves details such as the mouth.

5 LIMITATIONS

SCAIL-2’s end-to-end design depends on large-scale, high-quality paired data, while synthetic-data fidelity remains limited by generator capability and fine-grained positive samples are difficult to obtain.

  • The method fundamentally depends on large-scale, high-quality paired training data.Although the synthetic pipeline largely addresses data scarcity, the dependence on paired data remains.
  • Synthetic training-data fidelity hinges on the capability of the data generators.The pipeline reduces scarcity but cannot eliminate generator-quality limitations.
  • Reliable positive samples for fine-grained regions remain difficult to obtain despite Bias-Aware DPO modeling preference against bias.

6 CONCLUSIONS

SCAIL-2 presents an end-to-end character animation framework supported by a scalable synthetic dataset spanning diverse tasks. A DiT extracts and converts visual-context information, enabling broader zero-shot generalization and unified subtasks with a clear synergistic effect.

  • SCAIL-2 is an end-to-end framework for character animation.
  • A curated synthesis pipeline produces an end-to-end dataset spanning diverse animation tasks at scale.
  • A DiT with strong priors directly extracts and converts information from visual contexts, broadening zero-shot input generalization.
  • The end-to-end paradigm unifies several subtasks and yields a clear synergistic effect.

Supplementary Material · Appendix · A DETAILS ON THE MOTIONPAIR-60K DATASET

MotionPair-60K is a synthetic dataset of 59,376 end-to-end motion-transfer pairs, combining animation and replacement data with approximately a 3:1 ratio. Its training pipeline increases diversity through augmentation, pose-driven sampling, and a small subset of unreversed driving pairs used with synthetic denoising targets.

  • A DETAILS ON THE MOTIONPAIR-60K DATASET: MotionPair-60K contains 59,376 end-to-end motion-transfer pairs, with animation and replacement data in an approximately 3:1 ratio.The final composition and training sampling ratios are reported in Table 5.
  • A DETAILS ON THE MOTIONPAIR-60K DATASET: The source videos come from the same datasets as SCAIL, while LightX2V accelerates synthetic-data generation.The pipeline uses LightX2V for its generators.
  • A DETAILS ON THE MOTIONPAIR-60K DATASET: Random augmentations are applied to driving videos in animation mode during training.These augmentations are part of the dataset training procedure.
  • A DETAILS ON THE MOTIONPAIR-60K DATASET: Pose-driven animation is treated as a special case of end-to-end driving, with randomly sampled pose-driven pairs improving data diversity.This sampling strategy integrates pose-driven examples into training.
  • A DETAILS ON THE MOTIONPAIR-60K DATASET: End-to-end animation driving sequences use random cropping and stretching, whereas pose conditions use random skeleton scaling.The augmentations differ according to the driving modality.
  • A DETAILS ON THE MOTIONPAIR-60K DATASET: Figure 11 presents the distribution of MotionPair-60K’s data sources.The figure is specifically labeled as the distribution of data source.
  • A DETAILS ON THE MOTIONPAIR-60K DATASET: Approximately 5% of motion pairs retain the original driving direction and use the synthetic video ˜y directly as the denoising target.These pairs are sampled without reversing the driving direction.
  • A DETAILS ON THE MOTIONPAIR-60K DATASET: Table 5 reports MotionPair-60K’s composition, the additional pose-driven dataset, and their corresponding training sampling ratios.The table accompanies the dataset composition description.

B END-TO-END TRAINING DETAILS

End-to-end training fully fine-tunes the backbone for 3,500 steps with batch size 128 and learning rate 10−5, using varied resolutions and clip lengths on 64 H100 GPUs. Randomly replacing initial latent frames with conditional history latents enables autoregressive extension beyond the training clip length.

  • Long Video Generation: Randomly replacing the first two latent frames with conditional history latents enables autoregressive extension beyond the training clip length.This procedure follows Wan-Animate during training for long-video generation.
  • Implementation Details: The backbone is fully fine-tuned for 3,500 steps with batch size 128 and learning rate 10−5.This end-to-end training uses the stated optimization settings.
  • Implementation Details: Training uses spatial resolutions from 512×896 to 704×1280 and clip lengths of 65 or 81 frames.The model is trained across the specified resolution and temporal-length ranges.
  • Implementation Details: Training runs on 64 NVIDIA H100 GPUs for approximately one week using FSDP-2.The distributed training setup uses FSDP-2 across the stated hardware and duration.

C POST-TRAINING DETAILS · C.1 DETAILS OF PREFERENCE DATASET · C.2 BIAS-AWARE DPO IMPLEMENTATION

The post-training pipeline constructs preference data from synthesized videos and applies Bias-Aware DPO with regional hand-focused errors, an SFT anchor, and LoRA-only optimization. These choices target pose-estimation errors while stabilizing preference optimization.

  • C POST-TRAINING DETAILS: Bias-Aware DPO preference data are constructed through a dedicated pipeline overviewed in Figure 12.The figure presents the construction process for preference data used in Bias-Aware DPO training.
  • C.1 DETAILS OF PREFERENCE DATASET: Synthesized videos are treated as preferred targets, with accurate SDPose estimation producing clean positives and strict filtering applied to preserve fidelity.Less accurate estimation is used to amplify error propagation in negative samples, although the supplied passage is truncated before the full procedure.
  • C.2 BIAS-AWARE DPO IMPLEMENTATION: The DPO formulation optimizes a trainable flow-matching model vθ against a frozen reference model vref over preferred and dispreferred sample pairs.The relative flow-matching error between the trainable and frozen models defines the comparison used by the objective.
  • C.2 BIAS-AWARE DPO IMPLEMENTATION: Regional DPO restricts the preference objective to hand regions because pose-estimation differences are most significant in hand movements.This focuses optimization on the relevant discrepancy while avoiding distraction from other factors.
  • C.2 BIAS-AWARE DPO IMPLEMENTATION: A hand mask M is formed from the union of positive and negative hand bounding boxes per frame and downsampled directly into latent space.Masked velocity prediction errors replace the full-region score in the DPO loss.
  • C.2 BIAS-AWARE DPO IMPLEMENTATION: An SFT objective over positive samples is jointly optimized with DPO because DPO alone causes unstable training.The SFT term acts as an anchor preventing excessive divergence, with λ = 0.01 balancing the objectives.
  • C.2 BIAS-AWARE DPO IMPLEMENTATION: Post-training freezes the backbone and tunes rank-128 LoRA adapters for 400 steps at a learning rate of 1 × 10^-4 and batch size 24, with β = 5000.Only LoRA adapters inserted into transformer layers are optimized.

D EVALUATION DETAILS · E MORE EXAMPLES · G ETHICAL CONSIDERATIONS

The paper evaluates character animation with objective, human, and automatic perceptual metrics tailored to different scenarios, and reports robustness in challenging visual examples. It also frames SCAIL-2 for authorized use while acknowledging risks of deceptive or non-consensual impersonation.

  • D EVALUATION DETAILS: Self-driven evaluation uses PSNR, SSIM, LPIPS, and FVD as quantitative metrics.
  • D EVALUATION DETAILS: Human evaluation measures motion accuracy, identity consistency, physical plausibility, identity isolation, and environment integration across animation and replacement settings.These assess driving fidelity, reference-appearance preservation, physical constraints, character separation, and scene compatibility.
  • D EVALUATION DETAILS: VideoBench’s human-aligned automatic protocol evaluates Imaging Quality, Temporal Consistency, and Appearance Consistency on a 1–5 scale.The scale ranges from 1=very poor to 5=excellent.
  • D EVALUATION DETAILS: Studio-Bench contains 80 self-driven cases and 60/30/28 cross-identity cases for single-character, multi-character, and replacement tasks, with all evaluation cases excluded from training.
  • E MORE EXAMPLES: Additional visualizations show SCAIL-2 remains robust under challenging scenarios, including complex body shapes, fine-grained human-object interaction, and multi-character interactions.The examples emphasize character consistency, preservation of thin objects during interaction, and proper identity isolation.
  • G ETHICAL CONSIDERATIONS: Character animation and replacement can enable deceptive or non-consensual impersonation, so SCAIL-2 is intended for research and authorized creative use.
  • G ETHICAL CONSIDERATIONS: Users should obtain rights and consent for input identities and media and disclose generated or edited content when appropriate.Future releases may consider provenance metadata, watermarking, and misuse detection.
Loading 2606.10804v3…