PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization in the Loop

Benjamin Biggs, Oliver Boyne, James Charles, Andrew Fitzgibbon, Roberto Cipolla

arXiv:2007.11110v2cs.CV

TL;DR

The paper addresses the difficulty of reconstructing dogs in 3D across varied breeds and large test sets. It predicts dog pose and shape from monocular images using weak 2D supervision and an EM-refined shape prior, and reports improvements over competitive baselines, including those given ground-truth test annotations.

  • Problem

    Effective 3D dog reconstruction over large test sets remains un demonstrated, while dog breeds vary substantially in shape and texture.

    Source: §1 · p. 1

  • Method

    The method directly regresses dog pose and shape from a single image, using 2D training annotations and an EM-updated multi-modal shape prior.

    Source: §1 · p. 2

  • Results

    The method improves over competitive baselines, including when those baselines have access to ground-truth test data.

    Source: §5 · p. 13

  • Takeaways & Limitations

    The paper reports that tuning a Gaussian-mixture shape prior during training leads to improved reconstructions under weak 2D supervision from StanfordExtra.

    Source: §5 · p. 13

    The system has failure cases with multiple overlapping dogs and heavy motion blur.

    Source: §5 · p. 13

Abstract

from arXiv · show

We introduce an automatic, end-to-end method for recovering the 3D pose and shape of dogs from monocular internet images. The large variation in shape between dog breeds, significant occlusion and low quality of internet images makes this a challenging problem. We learn a richer prior over shapes than previous work, which helps regularize parameter estimation. We demonstrate results on the Stanford Dog dataset, an 'in the wild' dataset of 20,580 dog images for which we have collected 2D joint and silhouette annotations to split for training and evaluation. In order to capture the large shape variety of dogs, we show that the natural variation in the 2D dataset is enough to learn a detailed 3D prior through expectation maximization (EM). As a by-product of training, we generate a new parameterized model (including limb scaling) SMBLD which we release alongside our new annotation dataset StanfordExtra to the research community.

1 Introduction

Reconstructing dogs in 3D at scale is difficult because breeds vary greatly, while animal training data are harder to label and strong evaluation benchmarks have been scarce. The paper combines end-to-end monocular reconstruction with a learned shape prior and new model and dataset contributions aimed at diverse dogs in the wild.

1 Introduction

  • The method targets dog reconstruction across large test sets, where breed shape and texture vary substantially.The introduction notes that effective 3D dog reconstruction over large test sets had not yet been demonstrated.Source: §1 · p. 1
  • The work reconstructs dog pose and shape from single images without a fitting stage, uses 2D annotations only in training, fits a multimodal prior via EM updates, and adds explicit subpart scaling.Evaluation includes a test set of 1,703 low-quality internet images.Source: §1 · p. 2

1.1 Related work

  • The claimed task covers varied breeds, fur lengths, ear shapes and poses, and considerable occlusion.The paper contrasts these conditions with other systems’ higher test-time requirements, including manually supplied keypoints or silhouettes.Source: §1.1 · p. 2
  • SMB​​LD extends SMAL with scale parameters and a shape prior refined using a large image dataset to better represent diverse dogs.The paper describes this as an extension of the deformable quadruped model introduced from 41 artist-designed toy scans.Source: Monocular 3D reconstruction of human… · p. 4
  • Existing systems may require test-time energy optimization, which can fail with poor keypoint or silhouette predictions and adds computational burden.By contrast, the proposed method is trained from single in-the-wild images and requires no additional energy-based refinement.Source: Monocular 3D reconstruction of human… · p. 4
  • Earlier animal reconstruction evaluations were often qualitative or quantitative on fewer than 50 examples, motivating StanfordExtra as a large-scale benchmark.The paper presents the dataset as a way to support further progress in the field.Source: Monocular 3D reconstruction of human… · p. 4

2 Parametric animal model

The method builds on SMAL, a parameterized 3D animal mesh, and extends it to better represent the diversity of dog shapes. SMBLD adds scale parameters and an expanded shape prior.

2 Parametric animal model

  • SMAL parameterizes a deformable animal mesh with shape and pose, generating 3D vertices and joint positions.Shape coefficients describe an undeformed template; pose parameters govern joint rotations.Source: §2 · p. 5

2.1 Introducing scale parameters

  • SMAL captures quadrupeds, but its shape modes poorly represent dog variation; its construction used only four dogs.Source: §2.1 · p. 5
  • Examples show the mean SMBLD model alongside versions with elongated legs, tail, or ears.Source: Figure 3 · p. 6
  • SMBLD extends SMAL with six scale parameters, shared across joints where needed to avoid unrealistic independent scaling.The parameters scale mesh parts along local axes before pose is applied.Source: §2.1 · p. 6
  • The SMBLD shape prior is extended to include scale by fitting the model to 13 artist-designed 3D dog meshes.Source: §2.1 · p. 6

3 End-to-end dog reconstruction from monocular images

The paper reconstructs dog pose and shape from monocular images with a network trained using 2D annotations and shape priors. It adapts the shape prior with an EM loop using predicted shapes and in-the-wild training images.

3.1 Model architecture

  • The network predicts SMBLD pose and shape plus perspective-camera translation and focal length from a cropped monocular image.It predicts shape and scale parameters directly, using separate heads for blend-shape and scale parameters; log-scale prediction stabilizes early training.Source: §3 · p. 6, §3.1 · p. 7

3.2 Training losses

  • Training uses 2D keypoints and silhouettes instead of costly per-image 3D mesh annotations.Joint reprojection compares projected model joints with annotations; invisible joints are masked, while silhouette loss compares rendered and annotated masks.Source: §3.2 · p. 7
  • Pose and shape priors regularize predictions, and the authors report no need for a separate loss enforcing manually specified joint-angle limits.Source: §3.2 · p. 8

3.3 Learning a multi-modal shape prior.

  • A mixture of M Gaussian components replaces the unimodal shape prior to encourage more diverse predicted dog shapes.The unimodal prior tended to produce similar-looking 3D predictions across images of different breeds.Source: §3.3 · p. 8

3.4 Expectation Maximization in the loop

  • EM updates the mixture prior using predicted shapes and 2D labels from in-the-wild dog images, alternating updates to component parameters and per-image mixture weights.The initial artist-derived prior is described as unrepresentative of the real dataset’s shape diversity.Source: §3.4 · p. 8, The ‘E’ Step · p. 9

4 Experiments

The experiments introduce StanfordExtra and evaluate an end-to-end dog reconstruction method against optimization-based baselines, including on an unseen dataset. They also examine the contributions of the model components and show qualitative outputs across varied dogs and conditions.

4.1 StanfordExtra: A new large-scale dog dataset with 2D keypoint and silhouette annotations

  • StanfordExtra adds dog images with 2D keypoints and binary silhouettes, including substantial shape and pose variation, occlusion, interactions, and partial views.Mechanical Turk annotators provided a silhouette and 20 keypoints per image.Source: §4.1 · pp. 9–10, §20 · p. 10

4.2 Evaluation protocol

  • Evaluation uses 8,476 filtered images split by breed into 80% training and 20% test sets, measuring projected-silhouette IoU and joint PCK.Images were removed for mostly invisible keypoints or low annotation consistency.Source: §4.2 · p. 10

4.3 Training procedure

  • Training begins without silhouette loss or EM, then introduces silhouette loss, a mixture prior, and EM updates over 10 clusters.The stages run for 250 and 150 epochs, respectively, with EM every 50 epochs.Source: §4.3 · p. 10

4.4 Comparison to baselines

  • Our end-to-end method outperforms competitors given predicted keypoints and segmentations, and improves average IoU/PCK even against methods given ground-truth annotations.The authors also report faster test-time performance and improved results when their scale parameters and EM prior are added to 3D-M.Source: §4.4 · p. 11, Table 2 · p. 11

4.5 Generalization to unseen dataset

  • Without fine-tuning on Animal Pose, our method still improves over 3D-M using predicted keypoints and segmentations for that baseline.The evaluation uses the same data-preparation process as StanfordExtra.Source: §4.5 · p. 12, Table 3 · p. 13

4.6 Ablation study

  • Removing EM, the mixture prior, or scale parameters reduces overall performance; EM with the mixture prior improves metrics except ear keypoint accuracy.The authors attribute some ear-tip errors on extremely floppy-eared dogs to the model’s single articulation point per ear.Source: §4.6 · pp. 12–13, Table 4 · p. 13

4.7 Qualitative evaluation

  • Qualitative examples show system outputs for varied dog poses and shapes in challenging conditions, including Animal Pose dogs.Figure 5 compares our method, 3D-M, and CGAS using predicted meshes, canonical views, reprojected joints, and silhouette errors.Source: Figure 5 · p. 12, §4.7 · p. 13

5 Conclusions

The paper presents an end-to-end method for monocular 3D dog reconstruction, using weak 2D supervision and a richer learned shape prior. It reports improvements over competitive baselines and identifies failure cases and directions for future work.

5 Conclusions

  • The end-to-end method uses weak 2D supervision from StanfordExtra and tunes a Gaussian-mixture shape prior during training to improve reconstructions.The conclusion also describes the method as automatic and monocular.Source: §5 · p. 13
  • The method improves over competitive baselines, including when they receive ground-truth test annotations.Source: §5 · p. 13
  • Future work includes handling overlapping dogs and heavy motion blur, extending the EM formulation to video, and transferring learned dog knowledge to other species.Source: §5 · p. 13
  • The reported PCK results use the Yang and Ramanan definition normalized by 2D silhouette area.The note directs readers to the appendix for original tables and further details.Source: §5 · p. 13

6 Acknowlegements

The authors acknowledge support for computing resources, technical discussions, and project sponsorship.

6 Acknowlegements

  • The authors thank the GSK AI team for GPU-cluster access, four colleagues for technical discussions, and the GSK TDI group for project sponsorship.Source: §6 · p. 14

A Dataset curation

StanfordExtra annotations were collected from three independent workers per image, then processed to retain reliable keypoints and segmentation masks.

A Dataset curation

  • Each of the 20,580 Stanford Dog images was sent to three independent annotators for keypoint and segmentation-mask labeling, with a bounding box identifying the dog.Source: A Dataset curation · p. 17
  • Annotators marked 20 keypoints, including two per tail, three per leg, two per ear, nose, and jaw, and supplied visibility flags.Source: A Dataset curation · p. 17
  • Keypoint coordinates use the mean of clicks remaining after outlier removal when at least two remain; otherwise the point is marked invisible.Images with fewer than eight visible keypoints are removed from train and test splits.Source: A Dataset curation · p. 17
  • Segmentation submissions that fail simple criteria, such as minimum highlighted area, are sent for relabeling.Source: Segmentation · p. 17
  • Workers whose segmentation correlations with all other workers fall below a threshold are removed before the final mask is computed from remaining submissions.Source: Segmentation · p. 17
  • Figure I plots the distribution of segmentation-annotation accuracies across the entire labeled dataset.A worker’s accuracy is defined as the largest correlation with another worker’s segmentation.Source: Segmentation · p. 17, Figure I · p. 18

B Fitting SMBLD to 3D animation data

The appendix describes fitting SMBLD to varied 3D dog meshes to establish shape and scale priors, then using the resulting fits to initialize a mixture prior tuned during training.

B Fitting SMBLD to 3D animation data

C Training procedure

The training objective combines joint, silhouette, pose, shape, and mixture losses, and the method is trained in two stages. The first stage emphasizes joints; the second introduces silhouette alignment and periodic EM updates.

C Training procedure

  • The objective sums joint, silhouette, pose, shape, and mixture losses.Each loss term is weighted by a scalar λ.Source: C Training procedure · p. 19
  • Stage 1 trains for 250 epochs with Adam at a learning rate of 10^-4, weighting joint loss at 10.0 and pose and shape losses at 1.0 each.Silhouette and mixture losses are weighted at 0.0 in this stage.Source: C Training procedure · p. 19
  • Stage 2 introduces silhouette alignment and an EM update every 15 epochs, using 10 clusters selected by grid search over four cluster counts.The stage runs for 150 epochs with Adam at a learning rate of 10^-5.Source: C Training procedure · p. 19

D Probability of Keypoints Max (PCK-MAX)

The section evaluates reprojected 2D joint accuracy with PCK-MAX, a protocol that includes invisible ground-truth points. Tables 1 and 2 report results under this metric.

D Probability of Keypoints Max (PCK-MAX)

Loading 2007.11110v2…