Source-linked AI summary
Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture from Images "In the Wild"
Silvia Zuffi, Angjoo Kanazawa, Tanya Berger-Wolf, Michael J. Black
TL;DR
The paper tackles automatic 3D pose, shape, and texture capture for endangered zebras in challenging in-the-wild images with limited training data. It combines SMAL with synthetic-data training, learned shape variation, and texture prediction, enabling direct pixel-to-3D estimation and feature-based per-instance refinement.
Problem
Automatic animal pose, shape, and texture estimation is difficult because endangered species lack extensive 3D training data and appear camouflaged, occluded, and grouped in herds.
Method
The method integrates SMAL into an end-to-end regression network trained on synthetic images, learning shape variation and jointly predicting 3D pose, shape, camera, translation, and texture.
Results
The system directly regresses animal 3D pose, shape, and texture from pixels and uses predicted texture maps for per-instance network-feature optimization.
Takeaways & Limitations
Texture prediction supports more accurate pose and shape recovery, while the approach is presented as applicable beyond Grevy’s zebras.
Abstract
from arXiv · showhide
We present the first method to perform automatic 3D pose, shape and texture capture of animals from images acquired in-the-wild. In particular, we focus on the problem of capturing 3D information about Grevy's zebras from a collection of images. The Grevy's zebra is one of the most endangered species in Africa, with only a few thousand individuals left. Capturing the shape and pose of these animals can provide biologists and conservationists with information about animal health and behavior. In contrast to research on human pose, shape and texture estimation, training data for endangered species is limited, the animals are in complex natural scenes with occlusion, they are naturally camouflaged, travel in herds, and look similar to each other. To overcome these challenges, we integrate the recent SMAL animal model into a network-based regression pipeline, which we train end-to-end on synthetically generated images with pose, shape, and background variation. Going beyond state-of-the-art methods for human shape and pose estimation, our method learns a shape space for zebras during training. Learning such a shape space from images using only a photometric loss is novel, and the approach can be used to learn shape in other settings with limited 3D supervision. Moreover, we couple 3D pose and shape prediction with the task of texture synthesis, obtaining a full texture map of the animal from a single image. We show that the predicted texture map allows a novel per-instance unsupervised optimization over the network features. This method, SMALST (SMAL with learned Shape and Texture) goes beyond previous work, which assumed manual keypoints and/or segmentation, to regress directly from pixels to 3D animal shape, pose and texture. Code and data are available at https://github.com/silviazuffi/smalst.
1. Introduction
The paper addresses automatic 3D zebra pose, shape, and texture estimation in difficult in-the-wild images, where limited data, camouflage, occlusion, and herds challenge existing methods. It introduces an end-to-end approach using synthetic training data, learned shape variation, and texture prediction from pixels.
- Motivation: Grevy’s zebras are an endangered species with about 3000 individuals remaining, making extensive species-specific data collection impractical.The paper motivates automated capture as a way to study animal shape and pose without the investment available for human datasets.
- Challenges: Camouflage, overlapping herd members, similar appearance, and limited 3D data make direct transfer of human pose-estimation methods impractical for animals.These conditions particularly disrupt automatic segmentation and reliable keypoint extraction.
- Approach: The method overcomes scarce training data by combining real animal texture with synthetic variation in background, pose, illumination, camera, and shape.SMALR annotations from about 50 images support generating thousands of synthetic training images.
- Approach: A shared feature space links texture prediction to 3D pose and shape, while predicted image-to-texture flow handles the articulated animal’s disconnected texture regions.The network predicts pose, shape, and a texture map from a single image using a photometric training objective.
- Contributions: The approach learns a new shape space during training, estimates camera focal length, and supports test-time feature optimization without segmentation masks.The learned shape space is produced by a network layer rather than relying only on the original SMAL shape space.
- Evaluation: The method is evaluated quantitatively and qualitatively on 200 individual zebra images.The evaluation uses a zebra dataset described in the introduction.
2. Previous work
Earlier work provides strong tools for human pose and shape estimation, but animal shape and pose estimation remains comparatively limited. Existing animal methods often rely on manual annotations, simplified models, or single-animal reconstruction rather than scalable, realistic modeling across individuals.
- Human estimation: Human monocular pose and shape methods benefit from large scan-based body models, extensive datasets, joint detectors, and body-part segmentations.These resources are substantially richer than those generally available for animals.
- Appearance modeling: Prior human appearance models and bird texture methods motivate textured reconstruction, but the paper adapts appearance modeling to quadrupeds with more complex texture layouts.Zebra texture maps require handling articulated surfaces and spatially disconnected regions.
- Animal estimation: Existing animal methods commonly require manual keypoints or segmentation, or use simplified 3D primitives fitted from segmented images.These requirements limit fully automatic estimation from complex outdoor imagery.
- Animal estimation: Prior approaches lack realistic scan-learned animal models and generally estimate the shape of a single animal rather than variable body shapes across many subjects.The limitation concerns both fine detail and modeling intra-species variation.
- Animal models: SMAL introduced an articulated animal model for inter- and intra-species variation, while SMALR enabled fitting it to individual animals beyond the original shape space.The models were learned from toy scans and extended through multi-image fitting.
- Animal tracking: Animal tracking systems can estimate user-defined keypoints, but prior work had not addressed the 3D animal shape and pose estimation problem targeted here.The cited tracking applications include rodents, bees, and other small animals.
3. Approach
The approach regresses zebra pose, shape, texture, camera, and translation from images using SMAL-based models trained on digitally generated data. It learns a more expressive shape representation and predicts texture through stitched sub-images, with losses supervising masks, keypoints, geometry, flow, and appearance.
- Model-based regression: The method treats single-image zebra pose, shape, and texture estimation as model-based regression using the SMAL animal model.The network predicts texture, vertex displacements, 3D pose, translation, and focal length from an input image.
- Training data: Real animal appearance is rendered with variation in background, pose, illumination, camera, and shape to address limited and insufficiently realistic animal training data.The training set combines instance-specific models and captured textures with digitally generated variation.
- Training data: 12,850 synthetic RGB images were generated from 10 zebra models built with SMALR from 57 real zebra images.The generated images vary background, shape, pose, camera, and appearance, and include texture-flow and geometric annotations.
- Texture prediction: Texture is predicted as four sub-images that are stitched into a full texture map, addressing spatial discontinuities in the articulated quadruped layout.The framework also predicts uv-flow, which maps image pixels to texture-map elements.
- Shape prediction: The model learns shape variations through a network layer rather than relying only on the original SMAL shape space.Its vertex-displacement layer is initialized with SMAL blendshapes and optimized to represent more expressive shapes while exploiting symmetry.
- Training objective: The training objective combines mask, 2D keypoint, camera, image, pose, translation, shape, uv-flow, texture, and texture-consistency losses.The uv-flow loss is computed on visible textels because a single image provides only a partial texture-map assignment.
4. Experiments
The experiments evaluate direct network prediction, learned shape variability, and per-instance optimization against model-fitting and feed-forward alternatives. Results are reported on annotated zebra test images using pose, shape, and optimization comparisons.
- Evaluation design: The network is trained for 200 epochs with Adam and evaluated on 200 annotated test images, retaining the best validation model.The full model is compared with a version without the texture prediction module and its associated losses.
- Shape-space analysis: Shape variability is analyzed through 40 learned shape features, with the four highest-variance features used to visualize test-set deformations.The comparison also visualizes mean shapes from the network, SMAL, and SMALR.
- Per-instance optimization: Per-instance optimization runs for 120 epochs and retains the solution with the lowest photometric loss, with a comparison optimizing model variables instead of network features.Performance is reported in Table 1, while qualitative examples appear in Figure 8.
- Evaluation design: The study evaluates feed-forward prediction, texture ablation, noisy bounding boxes, model-variable optimization, and the proposed network-feature optimization.Table 1 also includes SMAL fitting requiring ground-truth keypoints and segmentations, plus prediction on synthetic data.
5. Conclusion
The paper presents automatic single-image capture of zebra pose, shape, and appearance in the wild without manual keypoints or segmentation. Texture prediction supports more accurate pose and shape recovery, while predicted textures enable per-instance feature optimization.
- Conclusion: The method directly regresses 3D pose, shape, camera, translation, and texture from in-the-wild images without keypoint detection or segmentation.The approach combines real appearance with synthesized pose, shape, and background variation.
- Qualitative results: Figure 6 presents input images, mesh overlap, predicted texture maps, 3D renderings, and 3D meshes in two columns.The figure organizes qualitative outputs for visual inspection of the reconstructed animals.
- Shape space: Figure 7 visualizes test-set shape variability and compares mean shapes from the network, SMAL, and SMALR.The reported variability is concentrated in the belly and ears.
- Conclusion: Predicting texture maps helps recover more accurate pose and shape.This conclusion is stated as an observed benefit of the texture-prediction component.
- Conclusion: Predicted texture maps enable per-instance optimization over encoder features using a photometric loss.The optimization is network-based and operates on individual instances.