Source-linked AI summary
Creatures great and SMAL: Recovering the shape and motion of animals from video
Benjamin Biggs, Thomas Roddick, Andrew Fitzgibbon, Roberto Cipolla
TL;DR
Animal shape varies widely within and across species, while obtaining diverse animal training data and reliable joint labels is difficult. The system identifies candidate 2D joint positions, links them into coherent skeletons through optimal joint assignment, and uses those skeletons to initialize generative model fitting for detailed per-frame shape and pose. The method performs well on real-world examples, generalizes to unseen animal species, and is robust to challenging input conditions.
Problem
Animal shape varies widely within and across species, while obtaining diverse animal training data and reliable joint labels is difficult.
Source: §1 · p. 1, Variability · p. 2Method
The system identifies candidate 2D joint positions, links them into coherent skeletons through optimal joint assignment, and uses those skeletons to initialize generative model fitting for detailed per-frame shape and pose.
Source: §1 · p. 1Results
The method performs well on real-world examples, generalizes to unseen animal species, and is robust to challenging input conditions.
Source: §5 · p. 15Takeaways & Limitations
Automatic segmentation supports 3D animal reconstruction without human intervention or prior knowledge of the animal species.
Source: §5 · p. 15The method assumes known camera intrinsics and processes multi-animal videos by segmenting each animal separately.
Source: §2.1 · p. 5, §3 · p. 6
Abstract
from arXiv · showhide
We present a system to recover the 3D shape and motion of a wide variety of quadrupeds from video. The system comprises a machine learning front-end which predicts candidate 2D joint positions, a discrete optimization which finds kinematically plausible joint correspondences, and an energy minimization stage which fits a detailed 3D model to the image. In order to overcome the limited availability of motion capture training data from animals, and the difficulty of generating realistic synthetic training images, the system is designed to work on silhouette data. The joint candidate predictor is trained on synthetically generated silhouette images, and at test time, deep learning methods or standard video segmentation tools are used to extract silhouettes from real data. The system is tested on animal videos from several species, and shows accurate reconstructions of 3D shape and pose.
1 Introduction
The paper targets automatic 3D animal shape and motion recovery from video, using silhouettes to connect learned 2D joint proposals with kinematic assignment and 3D model fitting. Its introduction frames this approach around species variability, scarce animal training data, and the need for a benchmark with real-animal annotations.
1 Introduction
- A deep hourglass network identifies candidate 2D joint positions, which optimal assignment links into skeletons that initialize a generative model-fitting back end to recover detailed shape and pose for each video frame.Source: §1 · p. 1
Variability.
- Animal tracking spans far greater shape variation across and within species than human tracking, while specializing to one species further limits already scarce training data.Surface texture also varies considerably between individuals.Source: Variability · p. 2
Training data.
- Animal joint labels are difficult to obtain consistently across species, and segmentation datasets are sparser than human data and shared across animal types.Some joints are easy to label, while others require trained operators to identify across species.Source: Training data · p. 2
- Because realistic synthetic RGB training images are difficult to generate, the method trains joint prediction on synthetic silhouette images instead.The SMAL model can generate quadruped shape and pose, but lacks surface texture maps.Source: Training data · p. 3
- The contributions include a silhouette-to-joint mapping, two OJA variants, a 3D fitting procedure, and BADJA, a dataset of eleven animal video sequences with sparse keypoints and silhouette segmentations.BADJA includes varied animal movement and challenging cases such as occlusion and motion blur.Source: Training data · p. 3
2 Related work
The authors situate dense 3D animal tracking among work on morphable models, silhouette-based recovery, and joint prediction, and believe theirs is the first dense 3D animal tracking in video without user-provided keypoints.
Morphable shape models.
- Earlier animal models either covered small object classes without a skeleton or used generalized cylinders fitted to hand-drawn skeletons.Combined skeletal and morphable models had also been used for hands and human tracking.Source: Morphable shape models · p. 4
- SMAL was built from 41 toy-figurine scans across five quadruped families and produced accurate single-frame fits to real animals despite its artists’ impressions.Earlier work depended on per-frame human-annotated keypoints, which the authors say would be costly and challenging for long sequences.Source: Morphable shape models · p. 4
Shape from silhouette.
- Prior work used silhouettes for single-view shape recovery, animal-gait extraction, and as a proxy for regressing 3D parameters.Source: Shape from silhouette · p. 4
- Animal joint prediction is limited by scarce annotations; Mathis et al. showed pose-estimation architectures can work in restricted animal domains, while this method trains on silhouettes.Related animal part-segmentation work uses general object-part datasets.Source: Joint position prediction · p. 4
2.1 Preliminaries
- The preliminaries define a mesh by pose and shape parameters, transform it into camera position, project its joints using known camera intrinsics, and render its silhouette differentiably.Source: §2.1 · p. 5
3 Method
The method turns segmented animal silhouettes into per-frame 3D shape, pose, and position estimates through a joint-prediction front end, optimal joint assignment, and generative model fitting. It uses multimodal training heatmaps and assignment losses to address ambiguous or missing joint predictions before refining the 3D fit.
3 Method
- The pipeline predicts multiple 2D joint candidates from silhouettes, assigns them into coherent skeletons, then fits a deformable 3D model to the silhouettes and joint candidates.Its test-time input is a sequence of binary silhouettes, and its output is shape, pose, and position parameters for each frame.Source: §3 · p. 6
3.1 Prediction of 2D joint locations using multimodal heatmaps
- Training heatmaps blend probability mass across leg-joint aliases to retain proposals when silhouette ambiguity confuses left, right, front, or back legs.The method uses a 0.75 weight for the joint and 0.25 for its alias, a ratio intended to pass a modest non-maximal suppression threshold without overly biasing the skeleton.Source: §3.1 · pp. 6–7
3.2 Optimal joint assignment (OJA)
- OJA selects one proposal or a null for each joint, balancing a learned skeletal prior, network confidence, and temporal coherence.The prior is a multivariate Gaussian over joint positions; a temporal term penalizes movement between frames, with τ controlling interaction between distant frames.Source: §3.2 · pp. 7–8
QP solution.
- Silhouette and bone coverage losses address overcounting by penalizing uncovered silhouette regions and bones that cross the background.The combined objective is optimized with a genetic algorithm; without coverage terms, it can be optimized as a quadratic program.Source: §3.2 · p. 8, QP solution · p. 9, GA Solution · p. 10
3.3 Generative model optimization
- Generative fitting minimizes silhouette, shape-and-pose prior, joint, and temporal energies to refine model parameters across the sequence.The joint energy stabilizes early nonlinear optimization and is scaled down as the silhouette term enters its convergence basin.Source: §3.3 · pp. 10–11
4 Experiments
Experiments assess joint prediction and 3D fitting on BADJA and synthetic sequences, report a 7% OJA-GA gain, and show visually comparable fits to SMAL without hand-clicked keypoints.
Datasets.
- BADJA contains 11 varied quadruped video sequences with 20 labeled 3D-mesh joints, visibility indicators, segmentation masks, and annotations every fifth frame.The sequences span species and movement types and include challenging visual scenarios.Source: Datasets · p. 11, Figure 6 · p. 12
4.1 Joint prediction
- The synthetic-silhouette-trained predictor generalizes to wild and unseen species and handles challenging poses and occlusions, but ambiguous front/back views remain difficult.The paper notes confusion when an animal faces directly toward or away from the camera.Source: §4.1 · pp. 11–12
4.3 Model fitting
- Staged fitting estimates camera from torso points, then optimizes shape, pose, and translation before adding the silhouette term late; rendered fits closely resemble inputs, including a camel.The silhouette term is introduced in the penultimate stage to avoid unsatisfactory local minima.Source: §4.3 · pp. 12–13
4.2 Optimal joint assignment
- Across BADJA sequences, OJA-GA improves joint prediction accuracy by 7% compared with raw network output, with coverage terms improving accuracy on most sequences.The horsejump high sequence is difficult because a human rider occludes part of the silhouette.Source: §4.2 · p. 12, Table 1 · p. 13
Comparison to other work.
- Visual fits are comparable to SMAL without hand-clicked keypoints; quantitative mesh error is assessed on synthetic sequences because reconstructed-mesh ground truth is unavailable.The synthetic evaluation reports mean distance between predicted and ground-truth vertices.Source: Comparison to other work · p. 13, Table 2 · p. 13, Figure 8 · p. 14
4.4 Automatic silhouette prediction
- An automatic pipeline uses DeepLabv3+ segmentation for 3D reconstruction from images without human intervention; missing interior contours, unseen species, and occlusion produce documented failures.The examples include facing-direction confusion, an elephant trunk interpreted as a tail, and heavy occlusion.Source: §4.4 · p. 14, Figure 10 · p. 15, Figure 11 · p. 15
5 Conclusions
The method reconstructs 3D animal shape and motion from video and is reported to generalize across species and challenging inputs without human intervention or prior species knowledge.
5 Conclusions
- The method reconstructs 3D animal shape and pose from video, with reported generalization to unseen species and challenging inputs, and operation without human intervention or prior species knowledge.The conclusion describes a quadruped model parameterized in shape and pose.Source: §5 · p. 15
6 Acknowlegements
The authors acknowledge technical discussions, IT support, the title pun, and proofreading.
6 Acknowlegements
- The authors acknowledge technical discussions, IT support, the title pun, and proofreading.They thank Ignas Budvytis, James Charles, Peter Grandi, Raf Czlonka, the Biggs family, and Philippa Liggins.Source: §6 · p. 15