Source-linked AI summary
MoZoo:Unleashing Video Diffusion power in animal fur and muscle simulation
Dongxia Liu, Jie Ma, Xiaochen Yang, Jiancheng Zhang, Bin Xia, Zhehan Kan, Nisha Huang, Jun Liang, Wenming Yang, Jin Li
TL;DR
Cinematic animal effects require labor-intensive, costly modeling of muscle and fur dynamics. MoZoo generates photorealistic animal videos from coarse meshes using multimodal guidance, with evaluations showing superior textural fidelity and spatio-temporal consistency across diverse species.
Problem
Traditional animal-effects production requires labor-intensive skeletal, muscle, and fur simulation, motivating methods that reduce this technical and resource burden.
Method
MoZoo combines a synthetic-to-real paired dataset pipeline with multimodal coarse-mesh video generation using Role-Aware RoPE and Asymmetric Decoupled Attention.
Results
MoZoo outperforms state-of-the-art methods in photorealism, temporal consistency, and multimodal synthesis flexibility across diverse species.
Takeaways & Limitations
MoZoo streamlines high-fidelity animal production by integrating muscle and fur dynamics into an end-to-end coarse-mesh-to-video process.
Takeaways & Limitations
The method remains limited for complex multi-animal interactions and severe occlusions, while broader hair and fur simulation is left for future work.
Abstract
from arXiv · showhide
The creation of cinematic-quality animal effects necessitates the precise modeling of muscle and fur dynamics, a process that remains both labor-intensive and computationally expensive within traditional production workflows. While generative diffusion models have shown promise in diverse artistic workflows, their capacity for high-fidelity animal simulation remains largely unexploited. We present MoZoo, a generative dynamics solver that bypasses conventional refinement to synthesize high-fidelity animal videos from coarse meshes under multimodal guidance. We propose Role-Aware RoPE (RAR-RoPE) which employs role-based index remapping to synchronize motion alignment while decoupling reference information via fixed temporal offsets. Complementing this, Asymmetric Decoupled Attention partitions the latent sequence to enforce a unidirectional information flow, effectively preventing feature interference and improving computational efficiency. To address the scarcity of high-quality training data, we introduce MoZoo-Data, a synthetic-to-real pipeline that leverages a rendering engine and an inverse mapping approach to construct a large-scale dataset of paired sequences. Furthermore, we establish MoZooBench, a comprehensive benchmark with 120 mesh-video pairs. Experimental results demonstrate that MoZoo achieves high-fidelity fur simulation across diverse animal skeletons and layouts, preserving superior temporal and structural consistency.
1 Introduction
MoZoo targets the labor-intensive, refinement-heavy process of producing lifelike animal effects by generating high-fidelity animal videos from coarse meshes under multimodal guidance. The paper introduces a synthetic-to-real dataset pipeline and benchmark to address limited paired data and evaluate fur simulation, temporal consistency, and structural consistency.
- Motivation: Traditional animal-effects workflows require labor-intensive skeletal rigging, muscle modeling, and high-fidelity fur synthesis to model anatomy and dynamics.Most development effort is concentrated in these physically-based refinement steps.
- Problem: Existing diffusion methods focus primarily on human subjects and consumer applications, leaving accurate fur-dynamics simulation largely unexplored because high-quality paired data are scarce.Generated videos also often contain synthetic artifacts and lack realism and fidelity.
- Data and benchmark: MoZoo-Data combines UE5-rendered paired coarse-mesh/RGB sequences with inverse-mapped coarse meshes from real-world animal footage to capture biological dynamics.The pipeline begins with diverse animal meshes and animations from public sources and is intended for public release with MoZooBench.
- Method: MoZoo bypasses traditional CG refinement by generating high-fidelity animal videos with realistic muscle and fur dynamics from coarse mesh inputs under multimodal guidance.Guidance can include text prompts, reference animal images, or videos.
- Evaluation: 120 mesh videos comprise MoZooBench, while experiments show detailed texture generation across different species with structural consistency and generalization across biological morphologies and motion patterns.The paper also reports significant improvements in photorealism, temporal consistency, and multi-modal synthesis flexibility over state-of-the-art methods.
2 Related Work
Related work spans diffusion-based video generation and reference-guided controllability, alongside graphics techniques for simulating hair and muscles. Despite advances in physics-based, data-driven, learned-prior, and anisotropic Gaussian approaches, achieving automated high-fidelity simulation remains challenging.
- Video Generation: Diffusion models have advanced large-scale video foundation models, while reference-based generation has become important for improving controllability.Early work extended text-to-image architectures to promote temporal coherence.
- Video Generation: Current reference-guided methods mostly use a single reference image to guide video generation.This passage identifies single-image guidance as the prevailing setup.
- Animal Simulation: High-fidelity hair and muscle simulation remains difficult because professional rendering pipelines require specialized tools, intricate shading models, and physically based path tracing.The geometric complexity and strand count also make manual modeling exceptionally time-consuming.
- Animal Simulation: Physics-based and data-driven methods, along with learned priors and anisotropic 3D Gaussians, have improved hair representation but have not resolved the trade-off between reconstruction automation and visual fidelity.The passage presents this trade-off as remaining elusive.
3 Preliminary
The video diffusion model encodes videos into latent representations and uses Rectified Flow with a transformer denoiser to learn data-to-noise velocities. During inference, it integrates the learned velocity from Gaussian noise and decodes the resulting latent into a video.
- Video diffusion formulation: Videos are mapped into latent representations z_0 by a video VAE encoder before Rectified Flow modeling.Rectified Flow linearly interpolates between noise and data during the forward process.
- Video diffusion formulation: The Diffusion Transformer v_θ regresses the target velocity u_t = ε − z_0 using a conditioning signal c, such as text prompts or reference images.Training minimizes the expected squared error between the predicted and target velocities.
- Video diffusion formulation: Inference starts from z_1 ∼ N(0, I) and solves d z_t = v_θ(z_t, t, c)dt from t = 1 to t = 0.The latent is updated iteratively according to z_{t−Δt} = z_t − v_θ(z_t, t, c)Δt.
- Video diffusion formulation: The generated latent is converted into the final video through a VAE decoder.
4 Method
MoZoo is a DiT-based video diffusion framework that synthesizes target animal videos from mesh and texture-reference videos. Its method combines paired-data construction with Role-Aware RoPE and Asymmetric Decoupled Attention to align modalities, preserve structural and fur attributes, and reduce interference and computation.
- Paired Data Construction: MoZoo-Data combines UE5 synthetic rendering with real-world video editing, mesh extraction, and quality filtering to produce 62K clips at 480P and 15 FPS.The synthetic pipeline uses diverse animal assets, scenes, and camera trajectories, while the real-world pipeline addresses the synthetic-to-real domain gap.
- Paired Data Construction: Training uses triplets of texture-reference, mesh-reference, and target videos, with the mesh and target sharing motion, facial expressions, and background dynamics.The texture-reference video supplies the target animal’s fur characteristics.
- Role-Aware RoPE: RAR-RoPE remaps token coordinates by role, using fixed temporal offsets for reference videos and distinct indices for reference images and masks.This avoids the spurious temporal prior imposed by standard 3D-RoPE and accommodates heterogeneous alignment requirements across input streams.
- Asymmetric Decoupled Attention: ADA partitions latents into target, mesh, mask, and reference segments, allowing target queries to aggregate multimodal information while restricting guidance branches to specialized interactions.The asymmetric masking establishes unidirectional guidance flow and reduces feature interference between mesh structure and high-frequency fur details.
- Asymmetric Decoupled Attention: Framewise mesh-key attention synchronizes each target frame with its corresponding structural key, while reference and mask keys provide global style and regional constraints.Reference and structural branches extract features within their own subspaces to prevent target noise from corrupting clean guidance signals.
5 Experiment · 5.1 Experiment Settings · Implementation Details
MoZoo is evaluated using a Wan2.1-T2V-14B-based video diffusion framework fine-tuned for multimodal inputs, with experiments conducted on the held-out MoZooBench benchmark against VACE and Recafade. The implementation uses standardized video sampling and multi-GPU fine-tuning settings.
- 5.1 Experiment Settings: MoZoo is built on the 14B-parameter Wan2.1-T2V-14B DiT-based video diffusion model.The model is selected as the base because of its superior performance.
- Implementation Details: All self-attention layers are fine-tuned to accommodate multimodal input conditions.This fine-tuning modifies the base diffusion model for the experiment’s multimodal setting.
- 5.1 Experiment Settings: Videos are resized to 480×832 and sampled as 41 frames at 15 fps.These preprocessing and sampling choices define the video inputs used during fine-tuning and evaluation.
- Implementation Details: Fine-tuning runs for 20K steps on 8 NVIDIA H200 GPUs with a learning rate of 2 × 10−5 and batch size 8.The reported training configuration combines the compute setup, optimization duration, learning rate, and batch size.
- 5 Experiment: MoZooBench is introduced as a quantitative benchmark whose data are excluded from the training set.The benchmark uses the same construction method as the real-world training dataset while preventing training-set overlap.
- Implementation Details: MoZoo is compared with VACE and Recafade, with reference images supplied because neither method supports controllable reference videos.The comparison is designed for the available controllable-input setting described in the passage.
Evaluation · 5.2 Quantitative Comparison
The evaluation measures video quality and reference alignment using eight quantitative metrics. Against existing baselines, MoZoo achieves higher structural fidelity and reference alignment for both static-image and dynamic-video references.
- Evaluation: The benchmark evaluates eight metrics across video quality and reference alignment.The metrics assess multiple dimensions of generated videos and source correspondence.
- Evaluation: Subject and background consistency measure identity preservation and environmental stability.These metrics evaluate whether the subject and scene remain consistent.
- Evaluation: Motion smoothness evaluates temporal coherence and detects jitter.It targets temporal behavior in the generated video.
- Evaluation: Imaging and aesthetic quality provide no-reference assessments of frame-level perceptual clarity and visual appeal.These metrics assess perceptual properties without requiring a reference.
- 5.2 Quantitative Comparison: Mozoo is compared with state-of-the-art methods using standard quantitative metrics.The comparison evaluates the framework against existing baselines.
- 5.2 Quantitative Comparison: For fairness, the VACE baseline uses the input video, a reference image, and a binary editable-versus-fixed-region mask.This conditioning setup specifies the information available to the baseline.
- 5.2 Quantitative Comparison: Mozoo consistently outperforms existing baselines in structural fidelity and reference alignment.The reported advantage holds across the quantitative comparison.
- 5.2 Quantitative Comparison: MoZoo achieves higher similarity to target sources with both static-image I2V and dynamic-video V2V references.The result applies to both reference modalities described in the comparison.
5.3 Qualitative Comparison
MoZoo is qualitatively evaluated for text-to-video and reference-to-video generation using mesh-guided motion, with static reference images used for fair comparison. It produces high-fidelity muscle and fur simulations, preserving intricate details and improving realism over prior methods.
- Evaluation setup: The evaluation compares text-to-video and reference-to-video generation, using static reference images for fair benchmarking against prior methods.The model can process dynamic reference videos, but the comparison uses static images.
- Text-to-video generation: MoZoo synthesizes realistic animal textures from text prompts and mesh-guided motion while preserving high-frequency details that appear over-smoothed in VACE.Its outputs retain intricate texture details and greater visual fidelity.
- Cross-category versatility: MoZoo delivers robust simulations across animal categories, capturing both muscle structures and complex fur textures with high fidelity.The qualitative results demonstrate versatility across different animal types.
- Text-to-video generation: For text-based generation, MoZoo improves textural depth and realism over VACE, whose fine-fur representations often appear flat.VACE can transform mesh-guided animal videos into specific species but struggles with detailed fur.
- Reference-to-video generation: In reference-based generation, VACE and Refacade simulate fur to some degree but exhibit clear limitations, including incoherent subjects from VACE.The supplied passage also notes that Refacade often struggles, although the sentence is truncated before specifying the issue.
5.4 Ablation Study
The ablation study evaluates input modalities and isolates the contributions of Role-Aware RoPE and Asymmetric Decoupled Attention, using video conditioning by default for structural analysis. Removing RAR causes anatomical drift, while disabling ADA compromises high-frequency detail reconstruction.
- Ablation setup: The study compares single-image and reference-video conditioning, adopting video conditioning as the default for structural ablations.This setup separates input-modality effects from the contributions of individual architectural modules.
- Architectural ablations: Removing Role-Aware RoPE causes spatial drift in anatomical features because reference textures are no longer precisely anchored to the target mesh.The ablation identifies explicit coordinate mapping as necessary for accurate reference-texture alignment.
- Architectural ablations: Disabling Asymmetric Decoupled Attention significantly compromises reconstruction of high-frequency details.The result highlights the decoupled attention mechanism’s contribution to fine-detail recovery.
Ablation on Architectural Components · Ablation on Reference Modalities
The ablations show that RAR and ADA preserve texture alignment and fine details, while video-conditioned reference generally surpasses image conditioning in identity fidelity, temporal stability, and texture consistency. Image conditioning remains comparable when poses align with the reference but degrades under substantial viewpoint variation.
- Ablation on Reference Modalities: Video conditioning consistently outperforms image conditioning across video quality and reference alignment dimensions.The comparison covers image-to-video and video-to-video input configurations.
- Ablation on Reference Modalities: Video-to-video conditioning achieves superior identity fidelity and temporal stability despite high-quality image-to-video results.The passage attributes the advantage to the video-conditioned configuration.
- Ablation on Architectural Components: Removing RAR and ADA causes texture misalignment and loss of fine details in V2V synthesis.The ablation specifically evaluates Role-Aware RoPE and Asymmetric Decoupled Attention.
- Ablation on Architectural Components: RAR and ADA support high-quality V2V results by preserving texture alignment and fine details.Figure 7 compares synthesis with and without these architectural components.
- Ablation on Reference Modalities: Image conditioning is comparable to video conditioning when the target pose closely aligns with the reference view.This equivalence holds under closely matched pose and viewpoint conditions.
- Ablation on Reference Modalities: Under extreme viewpoints absent from a static reference, image conditioning may omit textures or lose details, whereas video conditioning uses temporal cues to maintain high fidelity.The limitation appears when generated motion covers substantial angles not represented in the image reference.
5.5 More Applications
MoZoo enables zero-shot cross-species texture transfer, decoupling high-level biological textures from their original anatomy and mapping them onto different species. The transferred attributes remain biologically coherent and anatomically integrated, supporting photorealistic synthesis of novel biological entities.
- Cross-Species Texture Transfer: MoZoo decouples high-level biological textures from their original anatomical structures and maps them onto entirely different species.This enables cross-species texture transfer without preserving the reference animal’s original anatomy.
- Anatomical Coherence: Transferred surface attributes remain biologically coherent, appearing naturally integrated with the target species’ anatomy rather than applied as a superficial overlay.The method preserves anatomical coherence during texture migration.
- Novel Entity Synthesis: This zero-shot re-skinning paradigm supports efficient and versatile synthesis of high-fidelity, novel biological entities.The reported application extends synthesis beyond the original species and anatomical structures.
6 Conclusion
MoZoo transforms coarse mesh sequences into photorealistic animal videos by integrating muscle and fur dynamics end to end, reducing manual effort in high-fidelity character production. The framework addresses multi-modal alignment and structural-textural interference, while future work targets broader simulation and associated misuse risks.
- Contributions: MoZoo converts coarse mesh sequences into photorealistic animal videos by integrating muscle and fur dynamics in an end-to-end process.This streamlines traditional refinement workflows for high-fidelity character production.
- Contributions: Role-Aware RoPE (RAR-RoPE) addresses multi-modal alignment by eliminating artificial temporal priors.
- Limitations and Future Work: Future research will extend MoZoo to complex multi-animal interactions, severe occlusions, and universal hair and fur simulation across diverse character archetypes.The paper also identifies risks from generating unauthorized digital avatars or deepfake videos as the technology generalizes to human subjects.