MagicPony: Learning Articulated 3D Animals in the Wild

1. Introduction

• Learning 3D priors from abundant 2D images avoids hardware-heavy data collection, but requires learning the complex prior while reconstructing the training images.Collecting 3D data often involves multiple cameras, scanners, domes, and substantial manual effort.Source: §1 · p. 1
• Training uses single-view images, a 2D object segmenter, and skeleton topology, without requiring prior 3D shapes, keypoints, or viewpoints.At test time, the learned function estimates a new object's shape and texture from one image in a feedforward manner.Source: §1 · p. 1
• DINO-ViT features provide noisy correspondences for viewpoint inference, while a no-additional-cost multi-hypothesis scheme avoids local optima from greedy matching.The approach avoids requiring additional keypoint supervision.Source: §1 · p. 2
• MagicPony combines a canonical-space volumetric shape and appearance representation with an extracted mesh for posing and rendering.This hybrid retains advantages of neural rendering while enabling shape regularizers.Source: §1 · p. 2
• MagicPony achieves significantly better quantitative and qualitative results than prior work with less supervision, and generalizes to abstract drawings despite real-image-only training.The abstract-drawing examples demonstrate this generalization.Source: §1 · p. 2

Weakly-supervised Learning of 3D Objects

• Weakly supervised single-image methods often add geometric supervision such as 2D keypoints or template shapes to object masks; alternatives include part segmentation, video flow, or constrained bottlenecks.UMR uses part segmentations but produces coarse symmetric shapes, while DOVE uses temporal information and optical-flow supervision.Source: §2 · p. 2

Optimization from Multi-views and Videos

• Multiview NeRF methods use accurate cameras, while deformable-object video methods rely on engineered optimization and cues including optical flow, masks, or DensePose.LASSIE uses DINO-ViT features but learns a shared shape model from a small image collection, with per-instance articulation.Source: §2 · p. 3

Learnable Deformable 3D Representations

• Mesh-based deformation uses vertex offsets or constrained models such as cages and skeletal skinning; parametric models like SMPL and SMAL often require extensive 3D scans.Vertex-offset methods typically require heavy regularization such as ARAP.Source: §2 · p. 3
• Dynamic neural fields extend radiance fields with time-varying components, but inverse transformation from world coordinates to canonical space can be harder to learn than forward deformation.The paper proposes a hybrid SDF-mesh representation combining implicit-model expressiveness with mesh deformation and articulation.Source: §2 · p. 3

Viewpoint Prediction in the Wild

• Viewpoint prediction methods address local optima by modeling multiple possible viewpoints; U-CMR evaluates over 40 cameras per sample, whereas MagicPony explores one hypothesis per step at essentially no added training cost.MagicPony relates each instance to a jointly learned category-level prior rather than relying on a template shape.Source: §2 · p. 3

3. Method

• Training requires single-view images, object masks, and skeleton topology, and predicts shape, articulated pose, albedo, and lighting from one image.The viewpoint, shape, and appearance components are trained together using reconstruction losses.Source: §3 · p. 3

3.1. Implicit-Explicit 3D Shape Representation

• A neural SDF is converted differentiably into a mesh, combining implicit shape optimization with an explicit category template shared across instances.The template is initialized as an ellipsoid; DMTet extracts a watertight mesh from the SDF.Source: §3.1 · pp. 3–4

3.2. Articulated Shape Modelling and Prediction

• Instance-specific vertex offsets adapt the template, while blend skinning applies predicted bone rotations to produce articulated poses.Image features predict instance deformations and bone rotations; the posed shape models large, controlled articulation.Source: §3.2 · pp. 4–5

3.3. Appearance and Shading

• A Lambertian appearance model predicts canonical-space albedo and lighting, then renders shaded pixels and object masks for image and mask supervision.Albedo and lighting are predicted from image features; mask supervision includes a distance-transform term.Source: §3.3 · p. 5

3.4. Viewpoint Prediction

• A shared canonical feature field renders self-supervised ViT correspondences, while statistical multi-hypothesis viewpoint sampling explores alternatives using one hypothesis per iteration.The approach targets viewpoint local optima without rendering every candidate at each iteration.Source: §3.4 · pp. 5–6

3.5. Training Objective

• Each iteration samples one viewpoint, computes its reconstruction loss, then adds shape, deformation, articulation, and hypothesis regularizers for the model update.Viewpoint sampling uses the learned probability distribution.Source: §3.5 · p. 6

4.1. Data

• The horse training set combines 10.8k DOVE images with 541 images from three additional datasets; smaller giraffe, zebra, and cow datasets are used to finetune the horse model.Bird training combines DOVE and CUB images; the experiments use segmentation masks and the datasets’ original train/test splits where available.Source: §4.1 · p. 6

4.2. Technical Details

• The model uses eight subnetworks, freezes pretrained DINO-ViT weights, and trains the other components end-to-end from scratch for 100 epochs on horses and 10 on birds.Instance deformation and articulation are disabled for the first 30 horse epochs and 2 bird epochs; the viewpoint network uses a separate encoder because shared encoders made viewpoint learning unstable.Source: §4.2 · p. 6

4.3. Qualitative Results

• Single-image predictions produce detailed textured 3D meshes and articulated poses, and the model generalises from real images to paintings and abstract drawings.Abstract horse drawings require 100 iterations of albedo-only test-time finetuning; the horse model also generalises to giraffes, zebras, and cows after finetuning without other modifications, except disabling articulation for the first 5k iterations.Source: §4.3 · pp. 6–7, Figure 3 · p. 7, Figure 4 · p. 7

4.4. Comparison with Prior Work

• Compared with prior methods, MagicPony predicts more accurate poses and higher-quality articulated shapes; on horses it reconstructs all four legs while UMR predicts only two.For a fair texture comparison with UMR, the albedo network is finetuned for 100 iterations at test time.Source: §4.4 · p. 7, Figure 6 · p. 7, Figure 5 · p. 7
• MagicPony achieves significantly lower error on toy-bird shape reconstruction and significantly outperforms methods with stronger geometric supervision on keypoint transfer.The comparisons are reported in Tables 2 and 3, respectively.Source: §4.4 · p. 8, Table 2 · p. 8, Table 3 · p. 8

4.5. Ablation Studies and Limitations

• Without feature loss or multi-hypothesis viewpoint prediction, viewpoints collapse to frontal views and reconstructions become unnaturally stretched.The ablations examine each component and their joint removal.Source: §4.5 · p. 8, Figure 7 · p. 8
• Predicted texture may lack sufficient detail after one forward pass, and articulation requires a predefined topology that may vary by species.The authors identify automatic discovery of articulation structure from raw in-the-wild images as future work.Source: §4.5 · p. 8

• MagicPony reconstructs an object's shape, articulation, albedo, and lighting from a single image and generalises to abstract drawings.Source: §5 · p. 8
• The approach combines self-supervised representation learning with a new viewpoint sampling scheme.Source: §5 · p. 8
• MagicPony achieves significantly superior results to prior works, including those using more supervision.Source: §5 · p. 8

6.1. Additional Comparisons with Prior Works

• MagicPony predicts finer shapes and more accurate poses than prior works, whose viewpoint estimates often collapse to a limited range.Against LASSIE, it also produces more detailed shapes on unseen test images despite starting from a generic ellipsoid rather than a hand-crafted part-based shape.Source: §6.1 · p. 11

6.2. Ablation Studies

• Both feature reconstruction loss and multi-hypothesis viewpoint prediction are needed to prevent viewpoint collapse.The ablations examine these components quantitatively on Toy Bird Scans and show their effects on viewpoint distributions.Source: §6.2 · p. 11

6.3. Texture Finetuning

• Finetuning the predicted texture for 100 iterations improves quality, especially for images far from the training distribution.Textures shown for real horses in the main paper are single-pass predictions.Source: §6.3 · p. 11

6.4. Additional Qualitative Results

• Reconstructions generalize to abstract horse drawings, sculptures, toys, and animal categories including giraffes, zebras, and cows.The model also produces accurate single-image shape reconstructions across large variations in animal shape.Source: §6.4 · pp. 11–12

6.5. Failure Cases

• Texture prediction may not generalize beyond training textures; 100 albedo-network iterations can remedy this on paintings and abstract drawings.The finetuning takes less than 10 seconds.Source: §6.5 · p. 12
• Viewpoint prediction can fail on extreme or ambiguous views, often when DINO-ViT features are less distinctive.Source: §6.5 · p. 12
• From side views, the method may not distinguish an animal’s left and right legs using only masks and self-supervised DINO-ViT features.Source: §6.5 · p. 12

7.1. Articulation Model Details

• Bird skeletons use an eight-bone spine; quadrupeds add four three-bone leg chains connected from foot joints to nearby spine joints.The foot joints are identified as the lowest mesh points in four xz-quadrants.Source: §7.1 · p. 12
• Vertices receive skinning weights based on their distance to each bone, with the distance-softening temperature set to 0.5.Source: §7.1 · p. 12
• Bone rotations are limited to (−60°, 60°); quadruped twist and side-bending rotations are further limited to (−18°, 18°).These constraints are used to prevent unnatural poses.Source: §7.1 · pp. 12–13

7.2. Network Architectures

• The feature, template SDF, and lighting fields use five-layer MLPs; albedo and deformation fields use eight-layer MLPs, and articulation uses four transformer blocks.Coordinate inputs use sine and cosine encodings at eight frequencies.Source: §7.2 · p. 13

7.3. Hyper-parameters and Training Details

• Training enables articulation after 10k iterations and deformation after 40k, explores four viewpoint hypotheses initially, and takes roughly 20 hours for 150k iterations on one NVIDIA A40.Random sampling of viewpoint hypotheses is gradually reduced to 20%; the temperature decreases from 1 to 0.01 over 100k iterations.Source: §7.3 · p. 13

7.4. Keypoint Transfer Evaluation Details

• Keypoint Transfer evaluation uses PCK@0.1, with the correctness threshold set to 0.1 times the target object’s bounding-box maximum dimension.For PASCAL VOC Horse, the bounding box is padded by 5% on each side to match the cited implementation.Source: §7.4 · p. 13
• Ablations without feature loss or multi-hypothesis prediction show limited azimuth coverage; DOVE covers only part of horse azimuth, while UMR predicts frontal CUB views.The visualizations also compare Toy Bird Scan reconstructions with ground-truth scans.Source: Figure 10 · p. 14, Figure 11 · p. 14, Figure 12 · p. 15
• Additional visualizations show test-time texture finetuning, animatable and relightable horse reconstructions, abstract horse drawings and artefacts, and finetuned giraffe, zebra, and cow models.Source: Figure 14 · p. 17, Figure 15 · p. 17, Figure 16 · p. 18