PaperlayerView exact revision on arXiv

Source-linked AI summary

Kirin: Animal Motion Generation from In-the-Wild Video

Brian Nlong Zhao, Zhuoyang Pan, James M. Rehg, Jiajun Wu, Shangzhe Wu

arXiv:2609.01823v1cs.CV

TL;DR

Animal motion modeling lacks large-scale, high-quality data because controlled capture is impractical for most species. Kirin reconstructs motion from in-the-wild videos, builds AiM3D, and trains text-and-image-conditioned generation with automatic mesh animation. The framework reports state-of-the-art performance on in-distribution and external test sets.

  • Problem

    Animal motion modeling lacks large-scale, high-quality data because controlled capture is impractical for most species.

  • Method

    Kirin reconstructs 3D motion from in-the-wild videos, pairs sequences with captions in AiM3D, and trains a text-and-image-conditioned motion generator with automatic mesh rigging.

  • Results

    Kirin achieves state-of-the-art performance on in-distribution and external out-of-distribution test sets and produces animated 3D meshes through automatic rigging.

  • Takeaways & Limitations

    The dataset, generative model, and rigging pipeline form a unified solution for data-driven animal motion modeling and animation.

  • Takeaways & Limitations

    The shared generation model lacks animal-specific structural priors, which may limit species-specific realism across different anatomies.

Abstract

from arXiv · show

Understanding animal motion is fundamental to modeling animal behavior and biomechanics, yet progress in this area lags far behind human motion research due to the scarcity of high-quality motion data. While human motion can be captured in controlled environments, it is impractical for most animal species, resulting in small, domain-limited datasets that restrict downstream applications such as animation. To address this challenge, we introduce Kirin, a framework that reconstructs motion from video, learns motion priors at scale, and generates realistic motion that can be directly applied to animated assets. Using large collections of in-the-wild animal videos, we reconstruct 3D motion sequences and pair them with captions to create AiM3D, the first large-scale dataset offering aligned video-text-motion tuples for quadruped animals. Building on this dataset, we develop a visual-guided motion generation model that conditions on both text and image to guide the generation of realistic motion across diverse animal species. Finally, by leveraging an off-the-shelf image-to-3D model, we automatically rig and animate 3D meshes using generated motion, producing ready-to-render animated animals. Together, our dataset and framework establish a new foundation for large-scale, text and image conditioned animal motion generation and animation. Project page: https://kirin-ani.github.io/.

1 Introduction

Kirin addresses animal-motion data scarcity by reconstructing motion from in-the-wild videos, pairing sequences with captions, and learning text-and-image-conditioned generation and animation.

  • Motivation: Animal motion research is bottlenecked by scarce high-quality data because large-scale controlled capture is impractical for most species.This limits progress in behavior research, biomechanics, computer vision, and animation.
  • Motivation: In-the-wild videos offer diverse species, behaviors, and environments that controlled or manually authored datasets cannot fully capture.Recent 3D reconstruction advances make recovering motion from unstructured video feasible.
  • Kirin Framework: Kirin reconstructs 3D motion from large-scale animal videos and pairs sequences with text annotations to create AiM3D, the first large-scale aligned text–video–motion dataset for quadrupeds.The framework builds on AiM and uses SMAL-based reconstruction with VLM-generated descriptions.
  • Kirin Framework: The proposed motion generator conditions on both text and visual input, learning from in-the-wild videos rather than manually designed or synthetic motion.This targets broader and more diverse natural animal motion.
  • Results: Kirin achieves state-of-the-art performance on in-distribution and external out-of-distribution tests and produces animated 3D meshes through automatic rigging.The animation method is reported as more plausible and efficient than baseline approaches.

2 Related Works

Prior animal-motion datasets are limited by species coverage, ecological realism, motion availability, or skeletal consistency; Kirin combines video reconstruction and aligned captions to address these gaps.

  • Animal Motion Datasets: Animal motion data is scarce because most species cannot be captured naturally in laboratories, while anatomical and behavioral diversity complicates standardization.Existing controlled datasets cover few species and have limited ecological validity.
  • Kirin Pipeline: Figure 2 summarizes Kirin’s text-and-image-conditioned generation pipeline and its final rigging of generated motion onto an image-derived T-posed mesh.Frozen DistilBERT and DINOv3 encoders provide text and image features to the transformer decoder.
  • Animal Motion Datasets: Existing video datasets remain insufficient: BADJA has 11 3D-pose sequences, APT-36K has fifteen-frame clips, and AiM lacks 3D motion.AnimalKingdom provides single-image pose, while COP3D focuses mostly on stationary-pet orbit shots.
  • Animal Motion Datasets: Kirin reconstructs SMAL-based 3D pose from AiM videos and adds behavior-aware captions, yielding about 30k motion sequences paired with about 180k descriptions.The resulting dataset aligns video, motion, and text across diverse animal species.
  • 3D Animal Reconstruction: Model-based reconstruction provides explicit skeletal structure but may overfit 2D projections, whereas model-free methods are flexible but generally lack articulated skeletons.Kirin adopts AniMer initialization because accurate skeletal motion is prioritized over perfect shape.
  • Animal Motion Generation: Prior generation methods use single-species video, human or artist-created motion, or synthetic sources, limiting generalization to diverse natural animal movement.Ponymation is unconditional and horse-specific, while Puppeteer and MotionAvatar rely on synthetic motion.

3 Method

Kirin reconstructs quadruped motion from videos, learns text-and-image-conditioned motion, and transfers generated motion to rigged 3D assets. Its pipeline combines sequence-level SMAL refinement, separately tracked global translation, multimodal diffusion, and mesh retargeting.

  • Motion reconstruction: Kirin reconstructs complete 4D animal motion by refining SMAL pose sequences and combining them with separately estimated global translation.AniMer initializes per-frame pose, sequence-level refinement enforces keypoint alignment and temporal smoothness, and SpatialTrackerV2 supplies global translation.
  • Motion reconstruction: The reconstruction objective optimizes sequence-shared shape and per-frame joint rotations using projection alignment and temporal smoothness losses.The method uses confidence and visibility weighting for projected keypoints and geodesic rotation distances across frames.
  • Dataset: The dataset contains 29,979 reconstructed motion sequences and 179,874 textual descriptions generated from six descriptions per video.The test split contains 230 sequences following the benchmark subset.
  • Motion generation: The motion generator fuses DINOv3 image features, DistilBERT text tokens, and timestep embeddings within an MDM-compatible transformer.The fused conditioning is injected into every decoder block, and classifier-free guidance supports text-only, image-only, and text-plus-image inference.
  • Motion generation: The fused conditioning is intended to produce motions that follow textual actions while conforming to morphology supplied by the image.The image pathway addresses underspecified animal prompts involving species, breed, size, shape, coat, and viewpoint.
  • Asset animation: Generated motion is retargeted to a Rodin-reconstructed mesh through SMAL fitting, skinning-weight transfer, and linear blend skinning.The resulting assets are textured, rigged, and driven by synthesized animal motions.

4 Experiments

Kirin is evaluated for motion generation, reconstruction, and mesh animation against established baselines on in-distribution and external datasets. It consistently improves motion quality and alignment, while maintaining efficient reconstruction and producing more stable qualitative animations.

  • Quantitative Evaluation: AiM3D evaluation compares AniMo variants with Kirin using R-Precision, FID, MM-Dist, Diversity, and Multimodality over repeated trials.The evaluation generates 10 samples per text and reports means with 95% confidence intervals.
  • Motion Generation: Both Kirin variants outperform the baseline across all motion-generation metrics on the AiM3D test set.Retraining AniMo on AiM3D also improves performance, supporting the quality of the constructed dataset.
  • Motion Generation: Kirin outperforms AniMo across all metrics on the unseen AnimalML3D test set, indicating stronger generalization to external motion data.Neither the models nor the baseline were trained on AnimalML3D.
  • Motion Reconstruction: Kirin reconstruction achieves improved projection-based and 3D-aware metrics while processing at speeds comparable to near-real-time feed-forward baselines.The method includes post-optimization after feed-forward inference yet remains suitable for large-scale processing.
  • Qualitative Results: Compared with AniMo, Kirin better follows prompts and preserves consistent skeleton structure across generated skeletal-motion examples.The qualitative cases include coherent turning, correct rearing and landing, and stable bone lengths for mesh rigging.
  • Qualitative Results: Compared with Puppeteer, Kirin produces more stable and text-faithful mesh animations because it learns temporally smooth motion directly from real animal videos.Puppeteer often suffers from inconsistent generated videos, abrupt shot changes, or little motion on the input mesh.
  • Ablation Study: Image conditioning improves R-Precision, FID, and MM-Distance, with FID showing the largest gain, while Diversity and Multimodality decrease marginally.Visual cues strengthen perceptual realism and text–motion alignment but impose tighter spatial and appearance constraints.

5 Conclusion

Kirin unifies reconstruction, dataset construction, motion generation, and mesh animation for realistic 3D animal motion from in-the-wild videos. The framework achieves state-of-the-art performance on in-distribution and external test sets and supports applications including biomechanics, behavioral analysis, and character animation.

  • Conclusion: Kirin combines video-based reconstruction, the aligned AiM3D dataset, text-and-image-conditioned generation, and mesh rigging into one animal-motion framework.The pipeline produces animated 3D meshes from generated motion.
  • Conclusion: Kirin achieves state-of-the-art performance on both in-distribution and external test sets.The conclusion reports this outcome for the proposed motion-generation model.
  • Conclusion: The framework supports data-driven animal motion modeling and animation for biomechanics, behavioral analysis, and realistic character animation.These opportunities are stated as application areas enabled by the combined framework.

In-the-Wild Video

The supplied passage identifies this material as supplementary material, without describing in-the-wild video content.

  • The passage labels the material as supplementary material.
  • No video content, methods, or findings are described in the supplied passage.
  • The passage provides no section-specific information about in-the-wild video.

1 Dataset Details

AiM3D is derived from in-the-wild animal videos and pairs reconstructed motion with textual descriptions. Its examples visualize the progression from descriptions and video frames to reconstructed 3D articulation.

  • Dataset Details: AiM3D contains 29,979 motions, with 230 test motions and 29,749 training motions across 23 animal categories.Ten motions per category are assigned to the test set.
  • Dataset Details: Unlike manually authored animal-motion datasets, AiM3D derives motion directly from real in-the-wild videos.The dataset is built from video data rather than motions crafted by designers.
  • Dataset Details: Each visualization row presents two textual motion descriptions, video frames, and the corresponding reconstructed 3D motion.The layout exposes the alignment between descriptions, observations, and recovered articulation.

2 Caption Validation

Caption quality is evaluated with retrieval-based video-grounding metrics, using R Precision at top 1, top 2, and top 3 and comparison with AnimalKingdom.

  • Caption Validation: R Precision is measured at top 1, top 2, and top 3 to validate generated captions.The evaluation compares caption retrieval performance with the video grounding subset of AnimalKingdom.
  • Caption Validation: Table 4 reports video-grounding performance comparisons across datasets.

3 Dataset Distribution

AiM3D inherits animal-category and motion-type distributions from the AiM dataset, with reference distributions provided in the dataset figures. It is presented as a multimodal dataset aligning video, captions, and reconstructed motion.

  • Dataset Distribution: AiM3D follows the original AiM dataset’s distributions of animal categories and motion types.The category and motion-type distributions are shown in Figures 7 to 9 for reference.
  • Dataset Distribution: AiM3D provides aligned in-the-wild video, textual descriptions, and reconstructed motion sequences.The paper describes it as the only large-scale animal dataset simultaneously providing video, caption, and motion annotations, to the authors’ knowledge.
  • Dataset Distribution: Figure 7 shows the number of frames for each animal category.

5 VLM Prompts

The VLM captioning procedure uses Gemini 2.5 Flash with multiple prompts per video, covering concise descriptions, motion details, action characteristics, and global movement.

  • 5 VLM Prompts: Gemini 2.5 Flash generates textual descriptions for AiM videos using six prompts per video.A universal system message is used for the VLM captioning process.
  • 5 VLM Prompts: Each video is processed repeatedly with prompts that replace {animal} by specific animal categories.
  • 5 VLM Prompts: The prompts request short descriptions of the animal’s motion, action, detailed movement, speed, style, and action sequence.
  • 5 VLM Prompts: Figure 8 shows the number of videos for each animal category.
  • 5 VLM Prompts: One prompt explicitly asks for the motion type and global movement path.
  • 5 VLM Prompts: Prompt inputs include randomly selected in-context examples from AniMo4D and AnimalML3D for style reference.The example is inserted using the template “Following the style of this example: {example}”.

6 Global Translation Scaling Details

The global-translation scaling details include an equation based on crop size, camera focal lengths, and tracked-point depth at the first video frame. The reconstruction method is illustrated separately, with additional examples showing more evident global motion.

  • 6 Global Translation Scaling Details: The scaling expression uses the crop box size, camera focal lengths, and average depth of tracked points at the first frame.The cited prose defines crop size, focal lengths, and d_i^(0) as the depth of the i-th tracked point.
  • 6 Global Translation Scaling Details: Figure 9 shows the distribution of motion types across the full dataset, with each video assigned one to three motion labels.
  • 6 Global Translation Scaling Details: Figure 10 provides an overview of the motion reconstruction method.
  • 6 Global Translation Scaling Details: Figure 11 presents additional results with more evident global motion.The paper directs readers to the project page for interactive visualizations.

9 Image Condition

Image conditioning changes generated animal motion according to the input image, affecting both skeletal morphology and gait. The framework also has limitations from missing species-specific priors and reconstruction or rigging errors.

  • The same walking prompt produces motions with different skeletal variations and gait patterns when the input image changes.
  • Motion reconstruction infers articulation and global translation separately before combining them into the final motion data.
  • Additional examples include more noticeable global motion.
  • The shared generation model lacks animal-specific structural priors, which may limit realism across species with different scales and skeletal proportions.The paper suggests adaptive skeleton representations or category-aware conditioning as future directions.
  • Tail motion is not explicitly modeled because source videos and keypoint annotations often omit tail information, producing inconsistent or arbitrary tail dynamics.The limitation is especially relevant for species such as cats, where tails convey semantic and behavioral information.
  • Monocular reconstruction suffers from depth ambiguity, occlusion, limited viewpoints, and occasional rigging misalignment, which can yield physically implausible artifacts.Reported examples include foot sliding, floating, and incorrect limb trajectories.
Loading 2609.01823v1…