PaperlayerView exact revision on arXiv

Source-linked AI summary

3D Pose Estimation and 3D Model Retrieval for Objects in the Wild

Alexander Grabner, Peter M. Roth, Vincent Lepetit

arXiv:1803.11493v1cs.CVcs.AI

TL;DR

Retrieving 3D models from RGB images is challenging because real images and synthetic renderings differ. The paper estimates object pose and uses it to match RGB-image descriptors with descriptors of depth-rendered models; it reports improved pose estimation and accurate model retrieval from ShapeNet using unseen Pascal3D+ images.

  • Problem

    Comparing RGB images with 3D models is difficult because their representations differ and real images have a substantial domain gap from synthetic renderings.

    Source: §1 · p. 1

  • Method

    The approach predicts pose from virtual control points, then matches RGB-image descriptors to descriptors of depth images rendered from models under that pose.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    The method outperforms the state of the art for viewpoint estimation on Pascal3D+ and retrieves accurate 3D models from ShapeNet given unseen Pascal3D+ RGB images.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The approach supports category-agnostic predictions and offline pre-computed descriptors for scalable retrieval.

    Source: §5 · p. 8

    Depth-based retrieval captures geometry but not materials or textures, and retrieval becomes harder when pose estimation fails.

    Source: §4.2.2 · p. 8

Abstract

from arXiv · show

We propose a scalable, efficient and accurate approach to retrieve 3D models for objects in the wild. Our contribution is twofold. We first present a 3D pose estimation approach for object categories which significantly outperforms the state-of-the-art on Pascal3D+. Second, we use the estimated pose as a prior to retrieve 3D models which accurately represent the geometry of objects in RGB images. For this purpose, we render depth images from 3D models under our predicted pose and match learned image descriptors of RGB images against those of rendered depth images using a CNN-based multi-view metric learning approach. In this way, we are the first to report quantitative results for 3D model retrieval on Pascal3D+, where our method chooses the same models as human annotators for 50% of the validation images on average. In addition, we show that our method, which was trained purely on Pascal3D+, retrieves rich and accurate 3D models from ShapeNet given RGB images of objects in the wild.

1. Introduction

The introduction presents pose estimation as a prior for retrieving 3D models from RGB images, addressing representation mismatch, domain gap, and the cost of matching multiple renderings. It combines category-level pose estimation with matching RGB-image descriptors to descriptors of depth renderings.

1. Introduction

  • Retrieving 3D models from images is useful for scene understanding, augmented reality, grasping, and tracking, but comparing 2D images with 3D models is not straightforward.The two representations have considerably different characteristics.Source: §1 · p. 1
  • Real and synthetic RGB images differ in lighting, degradation, and backgrounds, while available model geometry and texture may not match real objects.These differences make direct comparison of rendered and real images challenging.Source: §1 · p. 1
  • Processing renderings from multiple poses is computationally expensive, yet object appearance varies with pose and mapping all poses to one descriptor does not scale to many categories.The proposed pose prior reduces retrieval to a single rendering per model.Source: §1 · p. 2
  • The pose estimator significantly outperforms the state of the art for viewpoint estimation on Pascal3D+ and supports category-agnostic predictions.It predicts virtual control points that generalize across categories.Source: §1 · p. 2
  • The retrieval method uses the predicted pose to render each 3D model once and matches RGB-image descriptors against descriptors of rendered depth images.The approach retrieves models from ShapeNet that represent object geometry in RGB images.Source: §1 · p. 2

2. Related Work

The related-work section surveys category-level 3D pose estimation and 3D model retrieval. It contrasts pose parameterizations and keypoint methods with classification, metric-learning, depth-based, and multi-view retrieval approaches.

2.1. 3D Pose Estimation

  • Pose-estimation methods range from 3-DoF angle regression or classification to full 6-DoF recovery from 2D/3D keypoint correspondences.Virtual control points offer an alternative to category-specific keypoints, which do not generalize across categories.Source: §2.1 · p. 2

2.2. 3D Model Retrieval

  • Model retrieval methods include fine-grained classification and metric learning, commonly matching real RGB images against renderings from multiple viewpoints.Other metric-learning approaches adapt features or combine information across views.Source: §2.2 · pp. 2–3
  • Depth-based retrieval avoids reliance on textured RGB renderings, while multi-view learning maps different input representations to a common space.Prior depth methods use real depth images, which are not available in many scenarios; the cited multi-view methods use voxels or LD-SIFT features.Source: §2.2 · p. 3

3. 3D Pose Estimation and 3D Model Retrieval

The pipeline estimates object pose from RGB images, then uses that pose to retrieve geometrically matching 3D models. It assumes roughly centered object windows and generally known categories, though pose estimation can work without category information with only a marginal loss of accuracy.

3. 3D Pose Estimation and 3D Model Retrieval

  • The method assumes known object windows and generally uses category labels, but reports only a marginal pose-accuracy loss when the category is unknown.The input window is roughly centered on the object and may come from a 2D detector.Source: §3 · p. 3

3.1. 3D Pose Estimation

  • The pose stage predicts 2D projections of the eight 3D bounding-box corners and recovers rotation and translation with PnP.Because PnP requires 3D control-point coordinates, the method predicts box dimensions and scales a unit cube to approximate them.Source: §3.1 · p. 3
  • The pose CNN jointly predicts corner projections and box dimensions, trained with projection, dimension, and regularization losses.The projection loss uses Huber distances to reduce sensitivity to inaccurate annotations.Source: §3.1 · pp. 3–4

3.2. 3D Model Retrieval

  • Retrieval renders candidate models as depth images under the estimated pose, reducing the need to process multiple poses while supporting untextured models without scene-lighting choices.Models are rescaled to fit the predicted 3D bounding box to improve alignment with the RGB image.Source: §3.2 · p. 4
  • Separate CNNs map real RGB and synthetic depth images to a common representation, with the descriptor loss trained using triplet loss.The real-image descriptor reuses hidden activations from the pose CNN, requiring no additional computation during pose-estimation inference.Source: §3.2 · pp. 4–5
  • Synthetic depth descriptors can be precomputed across the pose space, then matched at runtime using the viewpoint nearest the predicted pose.The authors describe this runtime procedure as fast and scalable while remaining accurate in their experiments.Source: §3.2 · p. 5

4. Experimental Results

The evaluation tests retrieval of ShapeNet models from unseen Pascal3D+ RGB images using a retrieval method trained solely on Pascal3D+, and separately assesses pose estimation on Pascal3D+.

4. Experimental Results

  • The retrieval evaluation uses unseen Pascal3D+ RGB images to retrieve 3D models from ShapeNet.The setup is presented as a realistic evaluation of retrieval for objects in the wild.Source: §4 · p. 5
  • The retrieval method is trained purely on Pascal3D+ data and evaluated on ShapeNet models.Source: §4 · p. 5
  • Pose estimation is additionally evaluated on Pascal3D+ because accurate pose estimation is essential to the retrieval approach.Source: §4 · p. 5

4.1. 3D Pose Estimation

The section evaluates 3-DoF viewpoint estimation on Pascal3D+, reports gains over prior methods, and tests category-agnostic prediction with a shared output layer.

4.1. 3D Pose Estimation

  • Evaluation follows [26], measuring geodesic viewpoint differences with MedErr and Acc π/6; the authors consider AVP unsuitable for this task.Acc π/6 is the percentage of viewpoint differences below π/6 (30°).Source: §4.1 · p. 5

4.1.1 3D Pose Estimation on Pascal3D+

  • ResNet with blur training improves Acc π/6 while maintaining low MedErr; ResNet also uses 24M parameters versus VGG's 135M.Blur training does not improve the VGG-based setup, and ResNet+blur is used in following experiments.Source: Table 1 · p. 5, §4.1.1 · p. 5
  • Small objects are a failure case: upscaling their image windows blurs them, and VGG performs poorly on the resulting over-smoothed images.Source: §4.1.1 · p. 5

4.1.2 Comparison to the State-of-the-Art

  • The approach significantly outperforms state-of-the-art methods in both MedErr and Acc π/6 across categories, with competitive results for individual categories.Table 2 compares viewpoint estimation using ground-truth detections on Pascal3D+.Source: Table 2 · p. 6, §4.1.2 · p. 6
  • Boat is especially challenging because of shape and appearance variability, low-resolution or obscured detections, and front–back ambiguity, yet the method still beats the state of the art.Source: §4.1.2 · p. 6
  • For tables, occlusion and ambiguous or symmetric orientations complicate evaluation; results are highly biased by only 21 validation samples.Source: §4.1.2 · p. 6

4.1.3 Category-Agnostic Pose Estimation

  • A single shared 19-neuron output layer enables category-agnostic pose prediction, which outperforms prior category-specific state of the art for some categories.The authors attribute some gains to shared information between similar categories, such as bike and mbike.Source: §4.1.3 · p. 6

4.2. 3D Model Retrieval

The section evaluates 3D model retrieval on Pascal3D+, tests retrieval from ShapeNet using unseen Pascal3D+ images, and demonstrates improved alignment of retrieved models with real-image objects.

4.2.1 3D Model Retrieval from Pascal3D+

  • 50% of validation images on average: with ground-truth pose, retrieval selects the same Pascal3D+ models as human annotators.The reported accuracy uses the ground-truth-pose setup.Source: §4.2.1 · p. 7, Table 3 · p. 7
  • Predicted-pose retrieval almost matches ground-truth-pose accuracy; random poses sharply reduce performance, while canonical views help train, bus, and tv monitor.Offline pre-computed descriptors retain accuracy while avoiding online rendering, using pose intervals of 10°.Source: §4.2.1 · pp. 7–8

4.2.2 3D Model Retrieval from ShapeNet

  • Across all twelve categories, ShapeNet retrieval yields accurate poses and models for unseen Pascal3D+ images, sometimes surpassing annotations, but depth retrieval omits materials and textures.Pose-estimation errors and heavy clutter are reported failure cases.Source: Figure 4 · p. 7, §4.2.2 · p. 8, Figure 5 · p. 8

4.2.3 3D Model Alignment

  • Using predicted 6-DoF pose and 3-DoF dimensions improves alignment over using predicted rotation alone, especially when detections are not fully accurate.Translation and dimensions support positioning and scaling the rendering.Source: §4.2.3 · p. 8, Figure 6 · p. 8

5. Conclusion

The conclusion presents pose estimation as a useful prior for 3D retrieval from real images and highlights the approach's scalability and ShapeNet results.

5. Conclusion

  • The approach maps real RGB images and synthetic depth images to a common representation to address the real–synthetic domain gap.Source: §5 · p. 8
  • The approach supports category-agnostic predictions and offline pre-computed descriptors, and outperforms the state of the art for viewpoint estimation on Pascal3D+.Source: §5 · p. 8
  • The method retrieves accurate ShapeNet models from unseen Pascal3D+ RGB images; the results motivate future research on joint learning from real and synthetic data.Source: §5 · p. 8

3D Pose Estimation and 3D Model Retrieval for Objects in the Wild Supplementary Material

The supplementary material previews additional qualitative retrieval results, failure-case analyses, and implementation discussions.

Supplementary Material

6. 3D Model Retrieval

Additional supplementary results show retrieval across twelve categories and improved alignment of retrieved models with imperfect object detections.

6. 3D Model Retrieval

  • For all twelve categories, additional results show accurate predicted poses and retrieved ShapeNet models on unseen Pascal3D+ validation images.Source: §6 · p. 11
  • For detections that are not fully accurate, predicted 6-DoF pose and 3-DoF dimensions improve alignment over using only a 3-DoF viewpoint.Source: §6 · p. 11
  • The alignment comparison uses an RGB rendering of the retrieved model and contrasts full-pose-and-dimension estimates with viewpoint-only estimates.Source: §6 · p. 11

7. Failure Modes

Pose-estimation failures cluster around low-resolution or ambiguous objects, while retrieval can also be disrupted by distortions, deformation, or occlusion. Some symmetry-related pose errors reflect ambiguity in the annotations and evaluation protocol, not just prediction difficulty.

7. Failure Modes

  • On low-resolution Pascal3D+ validation image windows, re-scaling produces over-smoothed RGB inputs lacking details and sharp discontinuities, leading to incorrect pose predictions that are difficult even for humans to identify.Source: §7 · p. 11
  • Heavy occlusion, poor illumination, and poses far from those seen during training also lead to incorrect pose predictions.Some examples remain difficult even when the annotated ground-truth pose is considered.Source: §7 · p. 11
  • Symmetric boats and tables can have multiple plausible poses, yet the evaluation protocol counts a prediction differing from the single annotated pose as a mistake.Examples include front–back ambiguity, azimuth rotations of π or π/2, and round tables with an axis of symmetry.Source: §7 · p. 11
  • Even after successful pose estimation, fish-eye distortion, deformed or demolished objects, and heavy occlusion can disturb model retrieval.The authors note that the ground-truth 3D models in these challenging examples are not accurate.Source: §7 · p. 11

8. Implementation Details

The implementation uses shared camera intrinsics, randomized data augmentation, and fixed training and loss settings. It normalizes model dimensions to a unit cube and reports high dimension-estimation accuracy across categories.

8. Implementation Details

  • The approach uses the same intrinsic camera parameters as the Pascal3D+ ground-truth pose computation.That ground-truth computation assumes identical intrinsic parameters for all images.Source: §8 · p. 11
  • Training augmentation jitters detections, adds Gaussian blur to sufficiently large windows, and randomly samples negative 3D models; occluded or truncated detections are excluded from evaluation.Augmentation parameters are randomized after each training epoch.Source: §8 · p. 11
  • Training uses batch size 50 for 100 epochs with Adam, lowering the initial 1e−4 learning rate by one order of magnitude after epochs 50 and 90.The projections are normalized to [0,1], and all 19 estimated values use a Huber loss with δ = 0.01; α = 1, β = 1e−5, γ = 1e−3, and m = 1.Source: §8 · p. 11
  • 3D dimensions are estimated in model space from [0,1] after normalizing models to a unit cube; the reported errors show high accuracy across categories.Table 4 reports mean performance across all categories.Source: §8 · p. 11, Table 4 · p. 11
  • The alignment illustration contrasts rotation-only rendering with rendering that uses the predicted 6-DoF pose and 3-DoF dimensions for an off-center detection.The qualitative retrieval figure also shows query images alongside ground-truth and retrieved-model renderings across twelve categories.Source: Figure 8 · p. 12, Figure 7 · p. 12
  • Illustrated failure cases include low-resolution windows, difficult illumination or pose, symmetry ambiguity, and retrieval under fish-eye distortion or occlusion.The captions describe these as qualitative examples rather than quantitative comparisons.Source: Figure 9 · p. 13, Figure 10 · p. 13, Figure 11 · p. 13
Loading 1803.11493v1…