Source-linked AI summary
4DEquine: Disentangling Motion and Appearance for 4D Equine Reconstruction from Monocular Video
Jin Lyu, Liang An, Pujin Cheng, Yebin Liu, Xiaoying Tang
TL;DR
Monocular equine 4D reconstruction is limited by incomplete observations and costly joint optimization of motion and appearance. 4DEquine separates these tasks using AniMoFormer, EquineGS, and two synthetic datasets, achieving state-of-the-art performance on real-world benchmarks and strong zero-shot generalization.
Problem
Existing monocular 4D methods struggle with incomplete animal observations, while optimization-based reconstruction is time-consuming and may require whole-video processing.
Method
4DEquine disentangles dynamic motion estimation from static appearance reconstruction, using AniMoFormer, EquineGS, the VAREN model, VarenPoser, and VarenTex.
Results
4DEquine achieves state-of-the-art motion recovery on APT-36K and AiM and superior full 4D reconstruction quality against cited optimization-based and general video-to-4D baselines.
Takeaways & Limitations
The framework provides an efficient monocular tool for equine 4D reconstruction and shows strong zero-shot generalization to donkeys and zebras.
Takeaways & Limitations
EquineGS requires keyframes in which the horse is not significantly occluded or truncated to reconstruct a consistent appearance.
Abstract
from arXiv · showhide
4D reconstruction of equine family (e.g. horses) from monocular video is important for animal welfare. Previous mainstream 4D animal reconstruction methods require joint optimization of motion and appearance over a whole video, which is time-consuming and sensitive to incomplete observation. In this work, we propose a novel framework called 4DEquine by disentangling the 4D reconstruction problem into two sub-problems: dynamic motion reconstruction and static appearance reconstruction. For motion, we introduce a simple yet effective spatio-temporal transformer with a post-optimization stage to regress smooth and pixel-aligned pose and shape sequences from video. For appearance, we design a novel feed-forward network that reconstructs a high-fidelity, animatable 3D Gaussian avatar from as few as a single image. To assist training, we create a large-scale synthetic motion dataset, VarenPoser, which features high-quality surface motions and diverse camera trajectories, as well as a synthetic appearance dataset, VarenTex, comprising realistic multi-view images generated through multi-view diffusion. While training only on synthetic datasets, 4DEquine achieves state-of-the-art performance on real-world APT36K and AiM datasets, demonstrating the superiority of 4DEquine and our new datasets for both geometry and appearance reconstruction. Comprehensive ablation studies validate the effectiveness of both the motion and appearance reconstruction network. Project page: https://luoxue-star.github.io/4DEquine_Project_Page/.
1. Introduction
4DEquine addresses incomplete-observation and optimization challenges in monocular equine 4D reconstruction by separating motion estimation from static appearance reconstruction. Its components and synthetic datasets support accurate motion, high-fidelity avatars, and strong benchmark performance.
- Framework: 4DEquine disentangles 4D reconstruction into dynamic motion recovery and static appearance reconstruction, bridged by the VAREN horse model.It recovers per-frame motion and generates Gaussian point clouds bound to VAREN mesh faces in a feed-forward manner.
- Motion reconstruction: AniMoFormer combines spatio-temporal processing with post-optimization to regress temporally consistent, pixel-aligned equine motion from video.VarenPoser supplies large-scale synthetic 4D VAREN annotations and diverse virtual camera trajectories for training.
- Appearance reconstruction: EquineGS reconstructs a high-fidelity, animatable 3D Gaussian avatar from a single representative image, addressing incomplete viewpoint coverage.VarenTex provides synthetic photorealistic multi-view images for training the feed-forward appearance network.
- Full framework: 4DEquine integrates motion and appearance reconstruction and can process videos of arbitrary length with a sliding-window strategy.The framework predicts Gaussian primitives from real images and animates them through the VAREN model.
- Evaluation: On unseen APT-36K and AiM equine videos, 4DEquine achieves state-of-the-art motion recovery and superior full 4D reconstruction quality over cited baselines.Reported motion metrics include PCK@0.05, PCK@0.1, Accel, and Chamfer Distance; ablations support both geometry and appearance designs.
2. Related Works
Related monocular 4D methods still struggle to recover complete animal geometry from incomplete observations, while some optimization-based approaches require costly per-scene training.
- 4D Reconstruction Methods: General monocular 4D methods capture static and dynamic scene geometry but often require per-scene training and fail to recover unseen surfaces.Feed-forward methods remove per-scene training requirements, yet the incomplete-observation problem persists.
- 4D Reconstruction Methods: Both generalizable and per-scene methods struggle to reconstruct full animal geometry without approximately 360-degree video around the target.This observation is presented as a barrier to real-world equine reconstruction applications.
- Equine Geometry Estimation: Animal reconstruction methods are broadly categorized as model-free or model-based, with this work focusing on model-based methods.The related-work discussion places equine reconstruction within this two-way categorization.
3. Preliminary
VAREN is a data-driven parametric horse model learned from scans of real horses, with muscle deformation that represents pose-dependent surface changes.
- The VAREN Model: VAREN is a 3D parametric horse model learned from thousands of scans covering 50 real horses across breeds and sizes.Its data-driven construction provides a horse-specific structural representation for reconstruction.
- The VAREN Model: Unlike SMAL, VAREN models non-rigid, pose-dependent surface deformation through regions corresponding to superficial horse muscles.The model therefore represents muscle-related shape changes rather than only a fixed articulated template.
4. Method
4DEquine separates equine 4D reconstruction into spatio-temporal motion recovery and single-image static appearance reconstruction. It trains these components with synthetic motion and multi-view appearance data, then connects them through VAREN-based deformation and rendering.
- Framework overview: 4DEquine uses AniMoFormer for spatio-temporal geometry recovery and EquineGS for feed-forward Gaussian avatar reconstruction.The two components are explicitly disentangled within the framework.
- VarenPoser dataset: VarenPoser supplies synthetic video clips with fitted VAREN poses, varied shapes and textures, and simulated fix, dolly, and orbit camera trajectories.The dataset contains 1171 clips at 512 × 512 resolution and 60 FPS.
- AniMoFormer: AniMoFormer processes frame-wise spatial features with a temporal transformer, then decodes pose, shape, and camera parameters over an N-frame window.Training uses fixed 16-frame windows, while inference supports arbitrarily long videos with sliding windows.
- AniMoFormer: AniMoFormer adds Post-Optimization that compares rendered masks and keypoints with pseudo ground truth to improve mesh-to-image alignment.The optimization combines 2D alignment, smoothness, pose regularization, and mask losses.
- EquineGS: EquineGS initializes 55,486 Gaussian positions from an upsampled VAREN mesh and fuses image, point, and global-context features through a dual-stream decoder.The decoder predicts positional offsets, rotations, scales, colors, and opacities for the Gaussian points.
- VarenTex dataset: VarenTex generates multi-view appearance training images by feeding VarenPoser-derived normal and canonical coordinate maps plus a ControlNet reference image into UniTex.The dataset addresses VarenPoser’s insufficient texture quality and video-only format for feed-forward avatar training.
5. Experiment
Experiments evaluate 4DEquine on real-world horse videos, unseen species, and ablations, covering motion, geometry, appearance, efficiency, and component effectiveness. The method shows strong reconstruction quality with single-image appearance inference and efficient video processing.
- Qualitative Evaluation: 4DEquine significantly outperforms baselines in qualitative pose and shape recovery, particularly in leg alignment accuracy.The improvement is attributed to smooth, accurate VAREN parameter regression combined with VAREN’s geometric detail.
- Qualitative Evaluation: 4DEquine recovers stable geometry and fuller appearance from limited viewpoints, while competing methods show distorted shapes, artifacts, floaters, or pose ambiguities.The comparison includes template-free, optimization-based, and generative methods on real-world inputs.
- Qualitative Evaluation: EquineGS generates novel-pose renderings from only the first video frame, producing a complete animatable avatar without optimizing over many frames.This setting directly contrasts with methods such as GART that use multi-frame optimization.
- Zero-Shot Generalization: 4DEquine produces plausible reconstructions for unseen donkey species despite training exclusively on horses with the VAREN model.For single-image testing, the input image is tiled into an N-frame stack.
- Quantitative Analysis: AniMoFormer achieves the highest PCK and the lowest Accel error and Chamfer Distance among evaluated baselines.These results indicate strong pose estimation, motion smoothness, and mesh similarity on the reported datasets.
- Quantitative Analysis: On AiM appearance evaluation, 4DEquine surpasses model-free methods perceptually and structurally, while fully optimized GART attains slightly higher PSNR.4DEquine has superior SSIM and LPIPS, outperforms Few-shot GART across all metrics, and leads all competitors on the zebra subset.
- Efficiency: 4DEquine reconstructs a video in 11 seconds per frame on one NVIDIA A100, whereas GART uses fixed 10k-step optimization taking 15 minutes.The reported 4DEquine time includes both geometry and appearance reconstruction.
6. Conclusion
4DEquine reconstructs equine motion and appearance by separating dynamic motion estimation from static avatar reconstruction, supported by two synthetic datasets. It achieves state-of-the-art performance and zero-shot generalization, while remaining limited by VAREN’s representation and environmental variation.
- 6. Conclusion: 4DEquine separates dynamic motion estimation from static appearance reconstruction through the VAREN model.AniMoFormer estimates motion, while EquineGS generates an animatable 3D Gaussian avatar.
- 6. Conclusion: AniMoFormer combines spatio-temporal transformer regression with post-optimization for smooth motion and 2D alignment.
- 6. Conclusion: EquineGS generates a high-fidelity, animatable 3D Gaussian avatar from a single image, avoiding costly per-video optimization.
- 6. Conclusion: VarenPoser and VarenTex provide synthetic motion and appearance data for training the two reconstruction components.
- 6. Conclusion: Experiments report state-of-the-art performance and strong zero-shot generalization to donkeys and zebras.
- Limitations and Future Work: The synthetic datasets inherit VAREN’s inadequate modeling of tail and mane physics and appearance, and the approach cannot account for dynamic lighting.
7. More Qualitative Results
Additional qualitative results visualize 4DEquine’s novel-pose animation on Internet video sequences using only each video’s first frame as input.
- 7. More Qualitative Results: The visualizations use Internet-collected video sequences and provide the first frame as 4DEquine’s input.
- 7. More Qualitative Results: The resulting figures show novel-pose animation rather than reconstruction of the input frame alone.
8. More Details about 4DEquine
4DEquine is trained on synthetic motion and appearance data, then applies sliding-window motion recovery and post-optimization to real videos. Supplementary figures document camera trajectories, qualitative outputs, and VarenTex samples.
- Training and Inference: AniMoFormer is trained on VarenPoser to regress VAREN pose and shape parameters, while EquineGS uses VarenTex to predict canonical Gaussian attributes.
- Training and Inference: At inference, AniMoFormer processes arbitrary-length monocular videos with sliding windows and post-optimizes pose and camera using 2D keypoints and masks.
- VarenPoser: VarenPoser assigns generated textures to 1,171 clips while limiting each unique texture to at most three clips.
- VarenPoser: The guidance set contains 500 Internet horse images used for appearance generation.
- VarenPoser: VarenPoser varies viewpoint initialization and samples fixed, orbit, and dolly camera trajectories with probabilities 0.4, 0.3, and 0.3.
- VarenPoser: Each VarenPoser frame is rendered at 60 fps and 512×512 resolution with VAREN parameters, 3D and 2D keypoints, visibility flags, and a segmentation mask.
- Qualitative Results: Figure 7 compares reference frames, AniMoFormer outputs, and final EquineGS reconstructions across time steps.
- EquineGS: The appearance model uses one selected input view and four additional views for supervision during training.
9. More Experiments
Additional experiments compare AniMoFormer with GART and AniMer, ablate temporal window size, and visualize challenging poses. AniMoFormer consistently outperforms the compared pose baselines, while longer windows improve metrics until memory limits.
- Pose Comparisons: AniMoFormer consistently outperforms GART across all pose metrics on AiM, even though GART is initialized with AniMoFormer estimates.
- Pose Comparisons: AniMoFormer outperforms AniMer + PO because of temporal attention and VAREN’s greater expressiveness than SMAL.
- Window Ablation: As window size N increases from 4 to 8 and 16, PCK@0.05 rises from 46.8 to 47.5 and 47.8.
- Window Ablation: As N increases from 4 to 8 and 16, acceleration error decreases from 27.0 to 25.8 and 25.7.
- Window Ablation: N = 32 causes an out-of-memory error, so sliding windows support inference on longer videos.
- Challenging Poses: Figure 10 presents challenging rearing and sitting poses to demonstrate 4DEquine’s robustness.
10. Failure Cases and Discussion
Severe truncation or occlusion exposes a limitation of EquineGS, whose appearance reconstruction can degrade in unseen areas. Keyframe selection therefore requires images in which the horse is not significantly occluded or truncated.
- EquineGS struggles to reconstruct a consistent appearance when input images suffer from severe truncation or occlusion.AniMoFormer can still reconstruct geometry in these cases because it uses occlusion and truncation augmentation during training.
- Keyframes supplied to EquineGS should avoid images where the horse is significantly occluded or truncated.The authors plan to address this challenge by efficiently fusing multiple keyframes.
- Figure 11 shows appearance degradation in unseen areas, contrasted with a reference frame depicting the horse’s true appearance.