ORMA: Optimization-based Monocular 4D Reconstruction of Articulated Animals

1 INTRODUCTION

• Animal model-based methods can fail on unseen species or poses, while model-free methods may produce temporally inconsistent or physically implausible results.The paper identifies combining model-based interpretability and stability with model-free adaptability as an open problem.Source: §1 · p. 2
• ORMA combines image-to-3D generative priors with SMAL+ articulation and optimizes animal motion in the world frame from video.It initializes shape from a reference frame, estimates video camera poses, registers the generated shape to SMAL+, and refines alignment with visual losses including dense feature correspondences.Source: §1 · p. 2
• Experiments on PAW4D and real videos with 3D ground truth show temporally coherent articulated geometry and motion in a consistent global frame.The authors suggest that applying ORMA at scale to in-the-wild videos could support large, diverse animal-motion datasets for learning motion models and synthesizing training data.Source: §1 · p. 2
• PAW4D adds a synthetic multi-species benchmark with ground-truth 3D geometry and camera motion, addressing the scarcity of 4D animal evaluation data.PFERD provides real horse videos with motion-capture-based 3D ground truth but is limited to horses.Source: §1 · pp. 2–3
• ORMA decouples instance-specific geometry from articulation, retaining parametric structure while adapting to diverse real-world animal forms and motions.The framework composes pretrained priors without task- or species-specific training and includes a unified objective for world-space geometry and motion recovery.Source: §1 · p. 3

3D Animal Reconstruction from Images

• Model-free methods offer morphological flexibility but typically lack explicit skeletal-pose control for structured behavioral analysis or biomechanical interpretation.Model-based approaches use predefined templates and can support tasks such as conformation analysis, health assessment, and motion tracking.Source: §2 · p. 3

4D Animal Reconstruction from Videos

• Video reconstruction methods range from model-free joint optimization of geometry, articulation, and appearance to model-based methods using structured animal representations.The surveyed methods include approaches focused on particular species as well as methods recovering global trajectories or building motion priors.Source: §2 · p. 3

Datasets for 3D and 4D Animal Reconstruction

• Many animal datasets provide image-space supervision, which cannot directly support or evaluate 3D shape and motion reconstruction.Existing temporal datasets include controlled multi-view or motion-capture recordings and in-the-wild videos; synthetic datasets also provide scalable supervision for geometry, motion, and camera dynamics.Source: §2 · p. 4

3.1 PRELIMINARY

• SMAL+ represents animal shape and pose with a deformable mesh and a richer learned shape space based on 145 registered 3D animal scans.Its formulation includes a template mesh, shape basis, joint regressor, and skinning weights.Source: §3.1 · p. 4

3.2 ORMA

• ORMA combines video-wide camera poses and depth maps with sparse, high-confidence DINO semantic matches to guide articulated pose optimization.The method initializes per-frame shape, pose, and camera with AniMer+, then uses MegaSaM for camera poses and SAM for animal masks. For shape adaptation, it fits SMAL+ to a scaled and aligned Hunyuan3D mesh, refines vertices with Chamfer optimization and ARAP regularization, and keeps the resulting shape fixed across frames.Source: §3.2 · pp. 4–5, Figure 2 · p. 5

3.3 LOSS FUNCTIONS

• With instance-specific shape fixed, the total objective combines appearance alignment, semantic correspondence, pose regularization, depth supervision, and temporal smoothing.The depth term constrains surface geometry and global 3D placement using MegaSaM estimates; the temporal term smooths translation, rotation, and motion.Source: §3.3 · pp. 5–6

3.4 ROBUST SEMANTIC CORRESPONDENCE

• DINOv3 features provide sparse, high-confidence local correspondences to address semantic ambiguity and instability in surface matching across frames.The method bakes features from multiple views onto the Hunyuan3D mesh, transfers descriptors to SMAL+, and searches local foreground neighborhoods for matches. It filters unreliable matches by a similarity threshold and softly weights valid matches, reducing the influence of unstable correspondences.Source: §3.4 · p. 6

• PAW4D targets the lack of 4D animal datasets with ground-truth 3D annotations, particularly in world coordinates.Source: §4 · p. 6
• The benchmark contains 115 videos: 51 dog, 47 fox, 13 puma, and 4 bear sequences.Sequences are rendered with follow, fixed, or orbit cameras, with randomized camera parameters and small perturbations for follow and fixed views.Source: §4 · p. 6
• Each video provides RGB frames, ground-truth masks, known cameras, and per-frame ground-truth meshes.Source: §4 · p. 6
• Figure 3 compares ORMA with four reconstruction methods on PAW4D and PFERD, contrasting image-only methods with video-based methods that use temporal information.Source: Figure 3 · p. 7

5 EXPERIMENTS

• Evaluation uses 26 PFERD videos and 109 PAW4D sequences, comparing four baselines across appearance, geometry, and trajectory metrics.Source: §5 · p. 7

5.1 COMPARISON

• Qualitative comparisons report improved shape fidelity with accurate pose and image alignment, and show global trajectories on PFERD and under varied in-the-wild capture conditions.Source: §5.1 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8
• With MegaSaM-estimated cameras, ORMA substantially improves 3D reconstruction on PFERD and achieves the best 2D appearance metrics and highest IoU3D on PAW4D.Trajectory RMSE is reported only for ORMA because the baselines do not recover camera-to-world transformations or global animal trajectories.Source: §5.1 · p. 8, Table 2 · p. 9

5.2 ABLATION

• Removing depth supervision lowers IoU3D from 0.414 to 0.313 and raises trajectory RMSE from 0.039 to 0.148, the largest reported ablation impact.Source: §5.2 · p. 8, Table 3 · p. 9
• Removing Hunyuan3D or replacing SMAL+ with original SMAL reduces reconstruction quality, while DINO and temporal smoothing improve alignment, 3D consistency, and trajectory accuracy.The full model achieves the strongest performance across all metrics.Source: §5.2 · p. 9

6 CONCLUSIONS

• ORMA combines generative 3D shape priors with SMAL+ articulation and geometric and semantic supervision to recover temporally coherent motion in a consistent world coordinate frame.Experiments report improved reconstruction accuracy and robust global motion reconstruction across diverse quadruped species.Source: §6 · p. 9
• PAW4D provides a synthetic multi-species benchmark with ground-truth geometry and camera motion for quantitative evaluation of 4D animal reconstruction.Source: §6 · p. 9
• ORMA requires SMAL+-compatible quadruped morphology; birds and aquatic animals are unsupported, and severe self-occlusion can hinder initialization.Accurate camera estimation is also required, while Linear Blend Skinning cannot fully capture strong pose- or motion-dependent non-rigid deformations.Source: §6 · p. 9

A.1 MORE DETAILS ABOUT PAW4D

• PAW4D addresses the shortage of reliable world-coordinate 3D ground truth, which prevents quantitative verification of global motion and errors such as scale drift.COP3D contains real-animal videos but lacks ground-truth geometry and camera calibration.Source: §A.1 · p. 15
• The source dataset contains 1,972 deformation sequences across 26 categories; filtering leaves 121 candidate animations suitable for terrestrial outdoor scenes.The filtering removes aquatic animals, physically implausible motions, and sequences shorter than one second.Source: §A.1 · p. 15
• Because the original textures were unavailable, the dataset uses EmbodyGen to synthesize multiple texture variations for each animal category.Source: §A.1 · p. 15
• Each animation is rendered with follow, fixed, and orbit cameras, and exports RGB, masks, depth, world-coordinate meshes, and camera calibration.Camera parameters are randomly sampled; follow and fixed cameras also receive small Gaussian jitter.Source: §A.1 · p. 15
• The PAW4D statistics table covers animation-sequence counts and generated texture variants.Source: Table 4 · p. 15

A.2 LOSS FUNCTIONS

• The 2D geometric loss compares rendered and observed edge maps using a Smooth-L1 penalty.Source: §A.2 · pp. 15–16
• The appearance loss combines Dice, perceptual, and boundary terms to align rendered silhouettes and boundaries with the observed foreground mask.The Dice term aligns overall foreground overlap.Source: §A.2 · p. 15
• The boundary losses complement silhouette overlap by matching overall edge magnitude and penalizing local contour misalignment.The edge-magnitude map is computed using Sobel gradients.Source: §A.2 · pp. 15–16
• MegaSaM depth estimates constrain both rendered-surface alignment and the animal’s camera-space root position.The surface and root terms provide local surface supervision and global 3D placement, respectively.Source: §A.2 · p. 16
• AniMer+ pose regularization limits implausible optimization drift, while temporal losses penalize acceleration in translation, global rotation, and root-relative joint motion.Source: §A.2 · p. 16

A.3 ROBUST SEMANTIC CORRESPONDENCE

• DINO features extracted from canonical views of the textured reconstruction capture details such as fur patterns and local markings for animal correspondence.Source: §A.3 · p. 17
• Projected view features are mapped back to the reconstructed surface and transferred to SMAL+, producing dense semantic descriptors that guide meaningful alignment.Source: §A.3 · p. 17
• DINO-based correspondences are more stable over time than CSE-based correspondences, improving reliability and reducing optimization drift.CSE predictions can vary across frames even under limited motion; DINO features remain more consistent, especially in articulated or visually distinctive regions.Source: §A.3 · p. 17