Source-linked AI summary
CrossPortrait: Cross-Species Portrait Animation via Adversarial Motion Alignment
Gezhong Pan, Zhe Li, Yunfan Tao, Yongheng Zhao, Shiqi Zhou, Juyong Zhang, Xiaosong Jia, Yanyong Zhang, Shuquan Ye, Shaofeng Zhang
TL;DR
Cross-species portrait animation must transfer motion between species with substantially different facial anatomy, while avoiding driving-species leakage. CrossPortrait combines geometric head motion, shared expression conditioning with species-routed adapters, and adversarial expression alignment. It reports strong cross-species comparisons and qualitative generalization to unseen source species when driven by trained species, while unseen driver species remain unexplored.
Problem
Cross-species portrait animation must transfer motion across substantially different facial anatomies, while expression features can entangle expression information with species cues that may degrade source-species fidelity.
Source: §1 · p. 2Method
CrossPortrait represents rigid head motion geometrically, uses shared expression conditioning with species-routed LoRA adapters, and applies Ex-ADA to suppress species-specific information in expression features.
Source: §1 · pp. 2–3, §3.2 · p. 5Results
CrossPortrait achieves the best animal self-reenactment metrics and highest A→H CSIM, while every valid baseline-over-CROSSPORTRAIT preference rate remains below 50% across cross-species settings.
Source: §4.2 · p. 7Takeaways & Limitations
A single shared generator supports cross-species animation across human, cat, and dog domains, with qualitative generalization to unseen source species when driven by trained species.
Source: §6 · p. 9Unseen-species driving and arbitrary facial topologies remain unexplored, and the absence of paired ground truth and validated cross-species motion metrics limits evaluation of motion accuracy and biological plausibility.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Cross-species portrait animation, which aims to animate a source portrait of species A using a driving video of species B, is particularly challenging due to substantial anatomical differences in facial structures across species. Previous works typically fine-tune human portrait animation models on animal video datasets but often suffer from driving-species leakage. To address this, we propose CrossPortrait, a novel framework for expressive cross-species portrait animation, supporting diverse scenarios including human-to-animal, animal-to-human and cross-species animal-to-animal (e.g. cat-to-dog) animation. Specifically, we decompose the portrait motion into rigid head poses and expression keypoints. To accommodate species-specific facial structures while facilitating cross-species expression alignment, we introduce a species-routed expression encoder that incorporates species-specific LoRAs into a unified expression encoder. Furthermore, we introduce expression-level adversarial domain adaptation (Ex-ADA) to suppress driving-species-specific information in the expression latents, thereby mitigating species leakage and promoting a shared expression space across species. Extensive experiments show that our method enables expressive and identity-preserving portrait animation across diverse species and outperforms other state-of-the-art approaches.
1 INTRODUCTION
Cross-species portrait animation must transfer driving motion across anatomically different species while preserving the source portrait’s identity and morphology. CrossPortrait combines shared expression alignment, species-specific adaptation, and adversarial suppression of species cues to address this challenge.
1 INTRODUCTION
- Cross-species animation spans human-to-animal, animal-to-human, and cross-species animal-to-animal transfer while preserving source identity, appearance, and species-specific morphology.The paper denotes a driving clip from species X applied to a source portrait of species Y as X → Y.Source: §1 · p. 2
- Species-specific facial anatomy makes expression alignment difficult, and leaked driving-species cues may degrade source-species fidelity.Head pose can use shared rigid transformations, whereas expressions vary substantially across species.Source: §1 · p. 2
- CrossPortrait uses a shared expression space and species-routed LoRA adapters to align expressions while capturing species-specific anatomical variation.The shared encoder backbone supports alignment, while species-specific adapters capture fine-grained patterns unique to each species.Source: §1 · pp. 2–3
- Ex-ADA trains a species classifier through gradient reversal to suppress species-specific identity cues in expression features and mitigate driving-species leakage.The resulting expression features condition video generation through cross-attention while the framework aims to preserve source-species fidelity.Source: §1 · pp. 2–3
- The framework represents rigid head motion with dense pose-flow and facial deformation with shared implicit-keypoint tokens across the three transfer settings.These contributions are presented as a unified dual-path framework for cross-species portrait-motion alignment.Source: §1 · pp. 2–3
2 RELATED WORK
Prior portrait-animation work ranges from explicit geometric controls to learned motion representations, with trade-offs between interpretability and flexibility. CrossPortrait combines geometric rigid-head motion with learned expression tokens and considers cross-species portrait transfer across human, cat, and dog domains.
Motion representations for portrait animation
- Explicit geometry offers a stable, interpretable motion interface but depends on predefined anatomy, while learned representations are more flexible but not inherently comparable across domains.CrossPortrait therefore models rigid head motion geometrically and facial deformation with learned implicit-keypoint tokens.Source: §2 · p. 3
Portrait animation beyond human faces
- Most portrait-animation methods focus on human-domain transfer; animal-driven portrait animation is less explored, and AnimateZoo studies full-body rather than portrait-scale motion.CrossPortrait covers human, cat, and dog sources and drivers, including human-to-animal, animal-to-human, and animal-to-animal settings.Source: §2 · p. 3
Adversarial domain alignment and domain-specific routing
- Domain-adversarial learning suppresses domain-predictive information through gradient reversal, while routed adapters provide domain-specific capacity around a shared model.CrossPortrait applies this combination to cross-species motion conditioning.Source: §2 · p. 3
- At inference, the driving-species label selects an expression-encoder LoRA, while the rigid-pose pathway, source-reference pathway, and shared generator remain species-agnostic.Ex-ADA is applied to source and driving expression features during same-species reconstruction training; only driving-side features condition motion at inference.Source: §2 · pp. 3–4
- Figure 3 describes Stage 1 as aligning expression representations across species while suppressing species-specific identity cues.The caption attributes these roles to the species-routed expression encoder and Ex-ADA.Source: Figure 3 · p. 4
3 METHOD
CrossPortrait separates rigid head motion from facial expression and learns a shared expression representation that accommodates species-specific anatomy while suppressing driving-species cues. It then conditions a video DiT on driving motion while retaining source appearance and morphology.
3.1 PROBLEM FORMULATION AND OVERVIEW
- CrossPortrait trains on same-species human, cat, and dog pairs to support cross-species animation while preserving source identity, appearance, and morphology.The framework has a motion-alignment stage followed by DiT-based video generation.Source: §3.1 · p. 4, §3.2 · p. 5
3.2 LEARNING CROSS-SPECIES MOTION ALIGNMENT
- A species-routed expression encoder adds species-specific LoRA components to a shared backbone and prediction head, accommodating anatomical differences within a common representation.The added capacity is intended to help interpret subtle species-dependent expressions.Source: §3.2 · p. 5
- Ex-ADA uses a three-class species classifier and gradient reversal to discourage species-specific information in expression features while preserving expression-related information through reconstruction and motion priors.The paper reports expression transfer with better preservation of the source animal’s species-specific oral morphology.Source: §3.2 · p. 5
3.3 DIT-BASED CROSS-SPECIES PORTRAIT ANIMATION
- The Stage 2 video DiT receives driving pose as motion-flow fields and expression features as temporally refined keypoint tokens, while a reference pathway supplies source appearance and morphology.Causal convolutions refine each keypoint trajectory before expression tokens condition cross-attention.Source: §3.3 · p. 5
3.4 TRAINING
- Stage 1 jointly optimizes motion encoders with reconstruction, motion-prior, and ADA losses; Stage 2 freezes those encoders and trains the DiT and motion-conditioning components.The warping and reconstruction modules are shared across species, and neither stage uses generator-side species routing.Source: §3.4 · p. 6
4 EXPERIMENTS
Experiments evaluate human-to-animal, animal-to-human, and animal-to-animal transfer, alongside animal self-reenactment and controlled ablations. CrossPortrait leads on reported self-reenactment metrics and A→H CSIM, while baselines receive under 50% preference across valid cross-species comparisons.
4.1 EXPERIMENTAL SETUP
- Evaluation covers H→A, A→H, and A→A transfer, with balanced cat–dog transfer as the primary trained-species A→A setting and unseen-source generalization assessed qualitatively.All evaluation identities and driving clips are held out; cross-species settings without paired ground truth use anonymized pairwise preferences.Source: §4 · p. 6, §4.1 · pp. 6–7
4.2 CROSS-SPECIES RESULTS
- CrossPortrait achieves the best self-reenactment metrics and highest A→H CSIM, while every valid baseline-over-CrossPortrait preference rate stays below 50% across cross-species settings.Tables 1 and 2 cover self-reenactment and cross-species preference results; lower baseline-over-CrossPortrait preference rates favor CrossPortrait.Source: Table 1 · p. 7, §4.2 · p. 7, Table 2 · p. 8
- Across bidirectional cross-species and cat↔dog examples, baselines show motion errors, geometric distortion, and species-specific leakage, while CrossPortrait better preserves source morphology and follows driving motion.Figure 5 distinguishes source portraits, driving images, and outputs, and marks visible artifacts and unsupported cases.Source: §4.2 · p. 8, Figure 5 · p. 8
4.3 ABLATION STUDIES
- The No-Alignment variant substantially weakens cross-species motion transfer and can produce unstable mouth geometry.It uses one shared ConvNeXt encoder for pose-flow and expression-token motion, removing species-routing LoRAs and Ex-ADA.Source: §4.3 · p. 8, Figure 6 · p. 9
- Removing Ex-ADA leaves more driving-species information in expression features, increasing species-specific leakage in H→A transfer.The adversarial suppression is applied to expression features rather than pose-flow or source-reference pathways.Source: §4.3 · p. 9
- Without the species-routed expression encoder, a shared encoder weakens fine-grained tongue-protrusion and mouth-opening motions.The routed encoder adds capacity for species-dependent anatomical variation while retaining a shared expression backbone.Source: §4.3 · p. 9
- Replacing causal temporal refinement with non-causal processing makes mouth and tongue motion less temporally consistent and lowers Expression and Overall preference rates.Future driving states can influence earlier expression tokens; perceived pose quality can also decline slightly during joint generation.Source: §4.3 · p. 9
5 LIMITATIONS
The evaluation is limited to human, cat, and dog drivers; unseen-species driving and arbitrary facial topologies remain unexplored. Missing paired ground truth and validated cross-species motion metrics constrain what preference and identity evaluations can establish.
5 LIMITATIONS
- Training covers human, cat, and dog drivers; unseen-species generalization is tested only qualitatively on the source side with trained-species drivers.Unseen-species driving and arbitrary facial topologies remain unexplored.Source: §5 · p. 9
- Cross-species animation lacks paired ground truth and validated cross-species motion metrics.Source: §5 · p. 9
- Preference-based evaluation and identity metrics cannot fully characterize motion accuracy or biological plausibility.Source: §5 · p. 9
6 CONCLUSION
CROSSPORTRAIT combines explicit geometric motion, learned expression conditioning, species-specific capacity, and expression-only alignment for portrait animation across human, cat, and dog domains. It supports diverse source–driving directions and improves motion transfer and source preservation over applicable baselines, while unseen driver species remain unexplored.
6 CONCLUSION
- CROSSPORTRAIT uses complementary pathways for explicit geometric motion and learned expression conditioning, with alignment applied only to expression features.A species-routed expression encoder retains species-specific modeling capacity, and a lightweight causal temporal module stabilizes expression-motion tokens before video generation.Source: §6 · p. 9
- The framework supports diverse source–driving directions and improves motion transfer and source preservation over applicable baselines.Ablations indicate the importance of balancing shared motion representations with species-specific modeling capacity.Source: §6 · p. 9
- Unseen source species generalize qualitatively when driven by trained species, but unseen driver species remain unexplored.The conclusion identifies broader species coverage and less structured motion domains as future directions.Source: §6 · pp. 9–10
Supplementary Material
The supplementary material includes method, evaluation, implementation, and data-protocol details. Its pose-flow section describes converting predicted rigid-pose parameters into dense motion fields and packing consecutive fields for video conditioning.
Supplementary Material
- The supplementary material lists sections on reconstruction objectives, evaluation metrics, baseline applicability, implementation, animal evaluation, and data and evaluation protocols.It also includes a human-to-human sanity check.Source: Supplementary Material · p. 14
- The contents list a same-species animal evaluation and a pairwise user-study protocol among the supplementary evaluation materials.The listed sections also cover training data and evaluation splits.Source: Supplementary Material · p. 14
A.1 POSE REPRESENTATION AND POSE-FLOW CONDITIONING
- PoseNet predicts pitch, yaw, roll, translation, and scale, which are converted into a dense three-channel displacement field.The field is the dense rigid pose-flow representation used for motion conditioning.Source: Figure 7 · p. 15
- Pose parameters are converted into dense rigid pose-flow fields, with four consecutive fields packed into 12 channels for projection to the VAE latent-control dimension.The projection uses a 1 × 1 × 1 Conv3D and produces 48 channels.Source: §A.1 · p. 15
A.2 ADVERSARIAL SCHEDULE AND DATASET SAMPLING
The supplementary sections specify the adversarial-training schedule, reconstruction-stage objectives, and evaluation metrics. They define expression and pose distances against driving frames and cosine similarity for source-identity preservation.
A.2 ADVERSARIAL SCHEDULE AND DATASET SAMPLING
- After a 3,000-step warm-up, Ex-ADA uses a gradient-reversal schedule, while human, cat, and dog data contribute approximately 30.9%, 34.6%, and 34.6% of the training mixture.Batches are shuffled from the mixture rather than explicitly class-balanced.Source: §A.2 · p. 15
A.3 STAGE-1 RECONSTRUCTION OBJECTIVE
- The Stage-1 objective combines reconstruction, perceptual, equivariance, keypoint, deformation-prior, global GAN, and Ex-ADA terms.Ex-ADA penalizes species-predictive information in expression representations through gradient reversal; Stage 1 has no temporal operation.Source: §A.3 · p. 15
B.2 AVERAGE EXPRESSION DISTANCE (AED)
- AED compares generated and driving expression vectors using mean absolute difference, with lower values indicating better agreement with the driving expression.The vector combines facial-expression, eyelid, and jaw-pose parameters predicted by SMIRK.Source: §B.2 · p. 16
B.3 AVERAGE POSE DISTANCE (APD)
- APD measures head-pose accuracy using SMIRK pose parameters, with lower values indicating better agreement with the driving head pose.The measure compares generated and driving pose vectors.Source: §B.3 · p. 16
B.4 COSINE SIMILARITY (CSIM)
- CSIM evaluates identity preservation by comparing ArcFace embeddings of generated frames and the source portrait, with higher values indicating better preservation.Cosine similarity is computed between the generated frame and source portrait embeddings.Source: §B.4 · p. 16
C.1 EVALUATION SETUP
The human-to-human sanity check evaluates self-reenactment with pixel-level metrics and cross-identity reenactment with unpaired motion and identity metrics. The applicability section distinguishes quantitative cross-species animal evaluation from qualitative examples using unseen source species.
C.1 EVALUATION SETUP
- The H→H sanity check uses 100 held-out human videos and compares CROSSPORTRAIT with six named animation systems plus Wan-Animate.Self-reenactment uses the first frame as source and remaining frames as ground truth; cross-identity pairs use different source and driving identities.Source: §C.1 · p. 16
- Self-reenactment reports PSNR, SSIM, LPIPS, L1, AED, APD, and CSIM, whereas cross-identity reenactment reports AED, APD, and CSIM without pixel-level ground truth.Table 4 and Figure 8 provide quantitative and qualitative comparisons, respectively.Source: §C.1 · p. 17, Table 4 · p. 17, Figure 8 · p. 17
D.1 APPLICABILITY MATRIX
- Cross-species animal-to-animal evaluation uses balanced cat↔dog pairs quantitatively, while unseen species appear only as sources in additional qualitative examples.The applicability matrix distinguishes source-side generalization from support for animal driving.Source: §D.1 · p. 17
E.1 STAGE-1 NETWORK ARCHITECTURE
Stage 1 separates pose and expression processing, adding species-routed expression branches and expression-only adversarial adaptation. Stage 2 conditions video generation on pose-flow and expression tokens, with training and evaluation protocols specified.
E.1 STAGE-1 NETWORK ARCHITECTURE
- PoseNet and ExpNet use separate ConvNeXt backbones; ExpNet predicts 21 shared 3D keypoint offsets and adds species-specific LoRA routes.The LoRA up-projection is zero-initialized, so routed branches begin identical to the shared backbone.Source: §E.1 · pp. 17–18
F.1 EVALUATION SETTING
- Table 5 reports method applicability, and animal-to-animal cross-identity evaluation uses pairwise human preference because no broadly accepted species-aware evaluator or paired ground truth is available.Failed applicable settings remain in the evaluation rather than being discarded.Source: Table 5 · p. 18, §E.1 · p. 18, §F.1 · p. 19
E.1 STAGE-1 NETWORK ARCHITECTURE
- The three-class species discriminator applies Ex-ADA only to the expression branch, leaving the pose branch unadapted.Its gradient-reversal layer is disabled for the first 3,000 optimization steps.Source: §E.1 · p. 18
E.2 STAGE-1 OPTIMIZATION
- Stage 1 combines reconstruction, perceptual, equivariance, keypoint-prior, deformation-prior, image-GAN, and Ex-ADA losses with separately configured weights.Training uses Adam and runs for approximately three days on four GPUs.Source: §E.2 · p. 18
E.3 STAGE-2 MOTION CONDITIONING
- Stage 2 packs four dense pose-flow fields into a 12-channel input projected to the 48-channel VAE latent control dimension, while expression offsets become temporally refined tokens.Expression cross-attention adapters are inserted every two transformer blocks, and expression conditioning is independently dropped with probability 0.2.Source: §E.3 · pp. 18–19
E.4 STAGE-2 OPTIMIZATION
- Stage 2 freezes the VAE and Stage-1 motion encoders while training the DiT backbone and motion-conditioning modules with a sigma-weighted MSE objective.Training uses AdamW on four GPUs with bfloat16 precision.Source: §E.4 · p. 19
G.1 TRAINING DATA
Both training stages use a shared human–cat–dog collection, and evaluation holds out source identities and driving clips. Pairwise user studies balance motion types and assess several distinct quality dimensions.
G.1 TRAINING DATA
- Both stages train on human portrait datasets and internally collected cat and dog videos, with approximately 15 hours of data per animal domain.Stage 1 uses 256 × 256 crops for PoseNet and ExpNet, while Stage 2 trains on 512 × 512 clips and resizes MotionNet inputs to 256 × 256.Source: §G.1 · pp. 19–20
G.2 EVALUATION SPLITS
- Evaluation holds out all source identities and driving clips and includes self-reenactment, cross-species transfers, cross-identity transfer, and a qualitative unseen-source study.The unseen-source study uses trained-species drivers, so it tests source-side generalization only.Source: §G.2 · p. 20, Figure 9 · p. 20
G.3 PAIRWISE USER-STUDY PROTOCOL
- Pairwise user studies balance five motion types and ask separate preference questions about expression, pose, source preservation, and overall visual quality.Ties count as half a vote when calculating preference rates.Source: §G.3 · p. 20