PaperlayerView exact revision on arXiv

Source-linked AI summary

SAM 3D Animal: Promptable Animal 3D Reconstruction from Images in the Wild

Xuyi Hu, Jin Lyu, Jiuming Liu, Yebin Liu, Silvia Zuffi, Liang An, Stefan Goetz

arXiv:2605.07604v1cs.CVcs.AI

TL;DR

Multi-animal 3D reconstruction is limited by scarce 3D data, species diversity, and occlusions. SAM 3D Animal jointly reconstructs prompted animals from one image using keypoints, masks, and Herd3D, achieving up to 54% AP and 80% mAP gains on Animal Kingdom.

  • Problem

    Multi-animal 3D reconstruction lacks diverse 3D datasets and methods handling inter-animal occlusions, spatial relationships, and pose dependencies.

  • Method

    SAM 3D Animal jointly reconstructs multiple animals from one image using SMAL+, keypoint or mask prompts, and set prediction trained with Herd3D.

  • Results

    Up to 54% AP gain and 80% mAP gain on Animal Kingdom, with prompts improving performance across all three benchmarks over the strongest baseline.

  • Takeaways & Limitations

    The framework provides a prompt-driven approach for reconstructing multiple animals in crowded and occluded scenes from a single image.

  • Takeaways & Limitations

    The method is mainly applicable to quadruped-like animals and may produce inaccurate relative spatial arrangements under severe occlusion.

Abstract

from arXiv · show

3D animal reconstruction in the wild remains challenging due to large species variation, frequent occlusions, and the prevalence of multi-animal scenes, while existing methods predominantly focus on single-animal settings. We present SAM 3D Animal, the first promptable framework for multi-animal 3D reconstruction from a single image. Built on the SMAL+ parametric animal model, our method jointly reconstructs multiple instances and supports flexible prompts in the form of keypoints and masks which enable more reliable disambiguation in crowded and occluded scenes. To train such a model, we further introduce Herd3D, a multi-animal 3D dataset containing over 5K images, designed to increase diversity in species, interactions, and occlusion patterns. Experiments on the Animal3D, APTv2, and Animal Kingdom datasets show that our framework achieves state-of-the-art results over both existing model-based and model-free methods, demonstrating a scalable and effective solution for prompt-driven animal 3D reconstruction in the wild.

1 Introduction

SAM 3D Animal addresses the challenges of multi-animal 3D reconstruction in occluded, diverse in-the-wild scenes with promptable, joint reconstruction using SMAL+ and keypoint or mask prompts. The proposed Herd3D dataset supplies over 5K multi-animal 3D training images, while experiments show consistent prompt-based improvements across benchmarks, including large gains on Animal Kingdom.

  • Motivation: Animal 3D reconstruction remains limited by scarce datasets, broad species variation, and inconsistent anatomical definitions, unlike rapidly advancing human-focused reconstruction.
  • Motivation: Existing animal methods typically reconstruct one animal at a time from pre-cropped inputs or strong detections, assumptions often invalidated by occlusion and interaction in multi-animal scenes.
  • Challenges: Multi-animal reconstruction requires resolving ambiguous instance association, jointly consistent pose and shape hypotheses, and dense per-instance 3D supervision.
  • Method: SAM 3D Animal is a promptable multi-animal framework that jointly reconstructs animals from one image using SMAL+ with optional keypoint and mask prompts.
  • Dataset: Herd3D contains over 5K images with per-instance ground-truth meshes, increasing diversity in species, interactions, and occlusion patterns for multi-animal 3D learning.
  • Results: Up to 54% AP gain and 80% mAP gain are achieved on the out-of-domain Animal Kingdom dataset with prompts, while performance improves consistently across all benchmarks.

2 Related Work

Prior animal 3D reconstruction methods are broadly model-free or model-based, while existing datasets remain limited in annotation scope and often focus on single-animal scenes. Promptable reconstruction has emerged in human mesh recovery to guide 3D estimation under occlusion and crowding.

  • Model-free reconstruction: Model-free methods learn 3D animal structure directly from image or video collections without assuming a predefined template.Early approaches separated predefined skeleton priors from instance-specific deformations, while later methods broadened species coverage through unified shape models or learned-shape skinning.
  • Model-based reconstruction: Model-based methods rely on predefined quadruped templates such as SMAL and fit or reconstruct them from 2D observations or images and videos.The parametric formulation provides interpretable and controllable representations.
  • Model-based reconstruction: AWOL maps CLIP-style embeddings to SMAL+ parameters, whereas RAW reconstructs animals with their environment but uses rigid assets rather than articulated models.Consequently, RAW does not address fine-grained articulated animal reconstruction.
  • Animal datasets: Animal dataset construction is difficult because animals are hard to capture in controlled environments and vary substantially morphologically across species.Existing benchmarks remain limited to 2D annotations, while existing 3D datasets predominantly focus on single-animal images.
  • Promptable reconstruction: Promptable mesh reconstruction uses auxiliary cues to guide 3D estimation under occlusion and crowding.Human-focused methods support spatial, semantic, keypoint, and mask prompts through promptable architectures or temporally consistent masklets.

3 SAM 3D Animal Model

SAM 3D Animal is a promptable, multi-animal reconstruction model built on the SMAL+ animal model. Its SAM-style decoder predicts multiple instances jointly, iteratively incorporates keypoint feedback, and uses bipartite matching for end-to-end training without NMS.

  • SMAL+ representation: SMAL+ provides a 145-dimensional shape space learned from 145 animals, including species such as giraffe, bear, mouse, and rat.The model represents shape with β ∈ R145, pose with θ ∈ R35×3, and global translation with γ ∈ R3.
  • Promptable multi-animal decoder: The SAM-style promptable Transformer predicts SMAL+ parameters, cameras, and bounding boxes for P = 30 possible animal instances at once, eliminating bounding-box input.The model supports masks or keypoints as prompts and can reconstruct all animals in an image without preprocessed bounding boxes.
  • Promptable multi-animal decoder: A layer-wise keypoint feedback loop refreshes 2D and 3D keypoint tokens using current predictions before subsequent decoder layers.2D updates combine positional embeddings with local image features, while 3D updates use geometric embeddings of normalized coordinates.
  • Multi-animal training: Hungarian bipartite matching assigns the fixed-size predicted hypotheses one-to-one to ground-truth animal instances, enabling end-to-end training without heuristic NMS.The matching cost combines multiple prediction terms, including bounding-box L1 distance.
  • Multi-animal training: The multi-task objective supervises SMAL+ parameters, 2D and 3D keypoints, and bounding boxes using parameter, keypoint, and localization losses.The bounding-box loss is Lbox = Lcoord + Lgiou + Lconf + Ldn, combining coordinate regression, GIoU, confidence, and denoising terms.

4 Herd3D Dataset

Herd3D is a large-scale dataset for multi-animal 3D reconstruction, containing over 5K images spanning 118 species. Its scenes contain 2 to 8 animals per image and include paired visual and 3D representations such as RGB, depth, canny maps, and reconstruction overlays.

  • Multi-animal scene composition: Each Herd3D image contains 2 to 8 animals, including generated multi-animal scenes with corresponding canny, depth, RGB, and 3D reconstruction representations.Figure 3 illustrates a generated scene with eight dogs and these corresponding modalities.
  • Dataset scale and diversity: Over 5K images and 118 species make Herd3D a large-scale dataset specifically designed for multi-animal scenes.The dataset is constructed to support multi-animal 3D reconstruction in real-world scenarios.

5 Experiments

Experiments evaluate SAM 3D Animal across in-domain and out-of-domain benchmarks using established baselines and 2D/3D accuracy metrics. Results show strong prompt-free performance, larger gains from keypoint prompts, improved robustness in challenging scenes, and benefits from Herd3D training.

  • Experimental setup: Training uses 49.2K annotated images, while evaluation covers Animal3D, APTv2, and the out-of-domain Animal Kingdom dataset.The benchmark corpus aggregates six datasets, including the newly introduced Herd3D.
  • Experimental setup: The method is compared with AniMer and GenZoo as model-based baselines and 3D Fauna as a model-free reconstruction baseline, using PA-MPJPE, PCK, AP, and mAP.AniMer uses SMAL, GenZoo uses SMAL+, and 3D Fauna represents model-free reconstruction.
  • Quantitative comparison: 80.7 mm PA-MPJPE and 49.3 mAP on Animal3D show that the prompt-free variant slightly outperforms AniMer’s 81.0 mm and 47.2 mAP.On APTv2, prompt-free PCK@0.1 reaches 87.9, surpassing GenZoo (64.1) and AniMer (62.4).
  • Prompt-driven performance: 57.4 AP and 89.0 PCK@0.1 on APTv2 are achieved with ground-truth keypoint prompts, compared with GenZoo’s 55.5 AP and AniMer’s 62.4 PCK@0.1.With ViTPose prompts, APTv2 AP rises from 49.4 to 55.5 and mAP from 23.5 to 27.9; Animal Kingdom AP rises from 45.0 to 50.5.
  • Qualitative comparison: Qualitative comparisons show that SAM 3D Animal better aligns reconstructions with observed pose and viewpoint than AniMer, GenZoo, and 3D Fauna across the benchmarks.The method also handles diverse species, unusual poses, viewpoint variation, and crowded scenes with several animals.
  • Ablation and visibility analysis: 57% relative mAP gain on low-visibility APTv2 samples shows that prompting matters most under heavy occlusion, while keypoint prompts dominate mask prompts and Herd3D improves all benchmarks.ViTPose prompting remains close to the ground-truth upper bound across visibility levels, and performance improves monotonically as keypoints increase from 0 to 15 with diminishing gains beyond 5.

6 Conclusion

SAM 3D Animal enables promptable joint 3D reconstruction of multiple animals from a single image, using keypoint and mask prompts to address ambiguity in crowded and occluded scenes. Its applicability is limited by the SMAL+ shape space and unconstrained relative depth ordering under severe occlusion.

  • Contribution: SAM 3D Animal jointly reconstructs multiple animals from a single image through a set-prediction formulation.This distinguishes it from prior methods that predominantly focus on animal-centric images.
  • Contribution: Keypoint and mask prompts resolve ambiguity in crowded and occluded scenes.The framework supports flexible prompting for multi-animal reconstruction.
  • Limitation: The SMAL+ shape space limits applicability mainly to quadruped-like animals.The paper identifies more flexible animal representations as a direction for future work.
  • Limitation: Unconstrained relative depth ordering can cause inaccurate spatial arrangements under severe occlusion.Future work could incorporate explicit depth-aware scene reasoning.

A Bipartite Matching Details

The method formulates prediction–target assignment as an optimal injection minimizing a weighted bipartite matching cost. The Hungarian algorithm computes the assignment, after which multi-animal predictions are reordered to match the ground-truth sequence.

  • Assignment formulation: Assignment searches for an optimal injection from M ground-truth targets to N predictions, with M ≤ N, minimizing the overall matching cost.Each prediction contains a bounding box, confidence score, and 2D keypoint locations.
  • Matching cost: The matching cost combines four weighted terms to promote localization precision and structural pose alignment.The terms include confidence, bounding-box, GIoU, and keypoint costs.
  • Matching cost: λconf = 1, λbbox = 1, λgiou = 1, and λkpts = 10 define the matching-cost weights.The keypoint term receives the largest specified weight.
  • Assignment resolution: The Hungarian algorithm computes the optimal permutation, then reorders multiple-animal predictions to align with the ground-truth sequence for subsequent supervisory losses.This alignment enables calculation of the standard supervisory loss functions.

B More Details about Herd3D

This section provides qualitative and statistical details of Herd3D, highlighting multi-animal scenes with two to eight visible animals and diverse species, poses, viewpoints, environments, and occlusion patterns.

  • Qualitative examples: Herd3D examples contain groups with two to eight visible animals.The number of visible animals gradually increases across representative multi-animal examples.
  • Dataset diversity: The examples cover diverse species, poses, viewpoints, background environments, and occlusion patterns.These variations characterize the diversity represented in Herd3D.
  • Annotation and visualization: Herd3D examples include structural cues such as edge maps, depth maps, and RGB information for annotation and visualization.The passage identifies these cues as corresponding annotation and visualization structures.

C Failure Cases in Herd3D Construction.

Herd3D construction encounters rendering failures in dense multi-animal scenes, especially with overlapping meshes and severe occlusions. These artifacts can distort animal geometry, pose, body parts, and instance boundaries.

  • Rendering failures: Large mesh overlap and severe inter-instance occlusion can prevent rendered images from faithfully preserving intended 3D geometry and pose.These failures were observed when rendering multi-animal scenes with Qwen-ControlNet.
  • Rendering failures: Animals facing away from the camera may be incorrectly rendered with forward-facing faces, producing inconsistent orientation.
  • Rendering failures: Small body parts can be misinterpreted, with ears rendered as noses or other facial structures.
  • Rendering failures: Spatially close animals may have blended body regions, causing one animal’s torso or limbs to be partially rendered onto another.
  • Rendering failures: Dense multi-animal scenes remain challenging for image-conditioned generative rendering under heavy occlusion and close physical interaction.

D More Experiments

The paper includes an ablation study examining how the number of prompt keypoints affects the method.

  • An ablation study evaluates the number of prompt keypoints.
Loading 2605.07604v1…