PaperlayerView exact revision on arXiv

Source-linked AI summary

Feature Mapping for Learning Fast and Accurate 3D Pose Inference from Synthetic Images

Mahdi Rad, Markus Oberweger, Vincent Lepetit

arXiv:1712.03904v2cs.CV

TL;DR

3D pose inference needs large training datasets, while manually labeling 3D training data is costly and difficult, requiring ad hoc estimation methods. The approach learns a mapping from real-image features to the synthetic feature space and uses the mapped features as input to a network predicting 3D pose. The method significantly outperforms the state of the art on LINEMOD object pose and NYU hand pose, with efficient inference compared with exemplar matching.

  • Problem

    3D pose inference needs large training datasets, while manually labeling 3D training data is costly and difficult, requiring ad hoc estimation methods.

    Source: §1 · p. 1

  • Method

    The approach learns a mapping from real-image features to the synthetic feature space and uses the mapped features as input to a network predicting 3D pose.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    The method significantly outperforms the state of the art on LINEMOD object pose and NYU hand pose, with efficient inference compared with exemplar matching.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that synthetic-to-real domain transfer can be achieved easily for 3D pose inference.

    Source: §5 · p. 8

Abstract

from arXiv · show

We propose a simple and efficient method for exploiting synthetic images when training a Deep Network to predict a 3D pose from an image. The ability of using synthetic images for training a Deep Network is extremely valuable as it is easy to create a virtually infinite training set made of such images, while capturing and annotating real images can be very cumbersome. However, synthetic images do not resemble real images exactly, and using them for training can result in suboptimal performance. It was recently shown that for exemplar-based approaches, it is possible to learn a mapping from the exemplar representations of real images to the exemplar representations of synthetic images. In this paper, we show that this approach is more general, and that a network can also be applied after the mapping to infer a 3D pose: At run time, given a real image of the target object, we first compute the features for the image, map them to the feature space of synthetic images, and finally use the resulting features as input to another network which predicts the 3D pose. Since this network can be trained very effectively by using synthetic images, it performs very well in practice, and inference is faster and more accurate than with an exemplar-based approach. We demonstrate our approach on the LINEMOD dataset for 3D object pose estimation from color images, and the NYU dataset for 3D hand pose estimation from depth maps. We show that it allows us to outperform the state-of-the-art on both datasets.

1. Introduction

Deep pose inference needs substantial labeled training data, while acquiring 3D annotations is especially difficult. The paper maps real-image features to the synthetic feature space before network-based pose prediction, and reports state-of-the-art results on object and hand pose benchmarks.

1. Introduction

  • Deep image inference often requires large training sets, and manually acquiring 3D labels is costly because human annotators cannot easily infer 3D data.The paper notes that 3D labels may require markers or semi-automatic annotation.Source: §1 · p. 1
  • Synthetic images offer abundant training examples, but their mismatch with real images can make synthetic-only training suboptimal.When some real images are available, prior transfer learning and GAN approaches address the domain difference.Source: §1 · pp. 1–2
  • Real-image features are mapped into the synthetic-image feature space and then supplied to a network that predicts 3D pose, extending feature mapping beyond exemplar-based retrieval.The mapping is learned using synthetic images rendered under poses matching available real training images.Source: §1 · p. 2
  • The method significantly outperforms the state of the art on LINEMOD object pose and NYU hand pose, with efficient network inference rather than exemplar comparisons.The figure contrasts baseline and proposed predictions for both datasets.Source: Figure 1 · p. 1, §1 · p. 2

2. Related Work

Related work addresses synthetic-to-real differences through synthetic training, domain adaptation, adversarial methods, and exemplar feature mapping. The paper positions its approach as a way to combine pose-matched synthetic imagery with direct network regression while avoiding some limitations of prior methods.

2. Related Work

  • Synthetic training images are easy to obtain from 3D models, but overfitting safeguards or extensive prediction refinement have been used, and the cited work reports better performance from combining real and synthetic images.[21] uses pretrained features and augmentation and was demonstrated on detection; [23] requires extensive refinement.Source: §2 · p. 2
  • Fine-tuning can overfit when target labels are scarce, while feature-distribution alignment methods include Maximum Mean Discrepancy (MMD) and domain-classifier approaches.MMD may align source and target features or learn a transformation into a common subspace.Source: §2 · p. 2
  • A layered transfer method constrains fully connected features with MMD, but is reported to suit discriminative clustering rather than regression and requires task-specific layer validation.The passage also notes the observation that feature transferability drops in higher network layers.Source: §2 · p. 2
  • The Siamese adaptation method [38] allows stream weights to differ while regularizing them, whereas this paper exploits same-pose synthetic renderings and reports outperforming [38].The passage describes [38] as adapting between very different domains.Source: §2 · p. 3
  • GAN image translation can produce visually similar images, but the cited comparisons report insufficient geometric precision for pose estimation, while adversarial features are described as unsuitable for regression.The passage relates the image-generation shortcoming to a lack of geometry in GAN models [50] [54].Source: §2 · p. 3
  • An assignment-based adaptation method depends strongly on semantically meaningful feature initialization, and its undefined class count makes it intractable for regression.Source: §2 · p. 3
  • The closest prior approach [29] maps real features to synthetic features for exemplar-based retrieval, which trades finer pose discretization and accuracy against slower nearest-neighbor search.The paper instead applies a learned mapping before a network that predicts pose.Source: §2 · p. 3

3. Approach

The approach trains a pose predictor on synthetic data while learning to map real-image features into the synthetic feature space. It examines how mapping changes feature differences and composes the networks for inference.

3. Approach

  • The model combines a feature extractor f, a pose predictor h, and a mapper g that transforms real-image features into the synthetic feature space.The method targets pose inference from real color images or depth maps.Source: Figure 2 · p. 3, §3 · p. 3, §3.1 · p. 3

3.1. Training

  • Joint training uses synthetic and real pose losses plus paired real–synthetic features to learn transferable features and map real features into the synthetic space.Each mapping pair contains a real image and a synthetic rendering of the target under the same 3D pose.Source: §3.1 · pp. 3–4
  • Rigid-object pose is represented by 2D bounding-box-corner reprojections, while hand pose uses normalized 3D joint locations.The rigid-object representation supports Euclidean loss and allows pose computation from reprojections using PnP.Source: §3.1 · p. 4

3.2. Effect of the Learned Mapping

  • Mean feature difference fell from 1.60 (σ=1.64) to 1.30 (σ=1.50) after mapping, which mostly removes large coefficient differences.The before-and-after feature differences are visualized in Figure 3.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3. Pose Prediction

  • At run time, the networks are composed to extract real-image features, map them to the synthetic feature space, and predict pose; the composition can be implemented as one network.Source: §3.3 · p. 4, §3.1 · p. 4

3.4. Network Details

  • The mapper uses two residual blocks on 1024-dimensional features; the object and hand tasks use different feature extractors and pose-predictor output sizes.The object predictor has 16 outputs, while the hand predictor has 42; synthetic-only pretraining improved convergence relative to random initialization.Source: §3.4 · p. 5

4. Experiments

The experiments evaluate feature mapping for 3D object pose estimation on LINEMOD and 3D hand pose estimation on NYU. They report strong benchmark results and fast inference across both tasks.

4.1. 3D Object Pose Estimation; 4.2. 3D Hand Pose Estimation

  • Training uses 5M online-rendered synthetic images for each task, with object views sampled across poses and hand poses perturbed with added articulations.Object renders use random ImageNet backgrounds; the synthetic hand depth maps do not contain the structured-light noise present in real maps.Source: §4.1 · p. 5, §4.2 · p. 7

4.1. 3D Object Pose Estimation

  • On LINEMOD, feature mapping outperforms the compared domain adaptation methods on the 2D Projection metric.The comparison includes transfer learning [38], GRL [12], and DDC [47], with the ground-truth 2D object center.Source: §4.1 · p. 5, Table 1 · p. 6
  • On LINEMOD, the method reports the best results from RGB images to date, using the refinement stage of [37] rather than the refiners used by and [23].The comparison includes 2D Projection, 6D pose, and 5cm 5° metrics; [23] reports only the 6D pose metric.Source: §4.1 · p. 5, Table 2 · p. 6
  • Using the same number of real training images, the method systematically improves LINEMOD results and needs about half as many images to match [37]'s accuracy.The comparison uses the 2D Projection metric.Source: §4.1 · p. 6, Figure 4 · p. 7

4.2. 3D Hand Pose Estimation

  • 7.4mm average Euclidean 3D joint error on real NYU test images improves the state of the art by 4.9mm, or almost 40%.Training on synthetic depth maps alone yields 21mm error on real test images.Source: §4.2 · p. 7, Table 3 · p. 7
  • On NYU, synthetic-only predictions are poor on real depth maps, while the method improves predicted poses, especially when depth maps are noisy.The qualitative comparison shows synthetic-only training on synthetic and real test maps alongside results from the proposed method.Source: §4.2 · p. 7, Figure 6 · p. 8
  • Nearly 40% better at a 20mm threshold, the method outperforms previous work on the fraction of frames with all joints within the distance limit.This is a difficult metric because a single erroneous joint can worsen a frame's result.Source: §4.2 · p. 7, Figure 7 · p. 8

5. Conclusion

The paper presents a simple, efficient approach that learns synthetic-to-real domain transfer for 3D pose inference and reports state-of-the-art results on object and hand pose datasets.

5. Conclusion

  • The approach learns a mapping between synthetic and real domains for 3D pose inference.The mapping is learned from synthetic images rendered under the same poses as available real training images, jointly with feature extraction and pose inference.Source: §5 · p. 8
  • The method is simple to implement, easy to optimize, and very efficient at run time.Source: §5 · p. 8
  • The approach outperforms the state of the art on popular datasets for 3D object and 3D hand pose estimation.Source: §5 · p. 8

6. Comparison to State-of-the-Art Baselines

The comparison evaluates the approach against strong object- and hand-pose baselines. The joint distributions indicate improvements on most images, often large on NYU and also present on LINEMOD.

6. Comparison to State-of-the-Art Baselines

  • The numerical comparison uses BB8 and DeepPrior++, identified as the second-best-performing methods in the main paper.Figure 8 provides detailed numerical comparisons on both datasets.Source: §6 · p. 11
  • The joint distributions show improved pose estimates on most images compared with BB8 on LINEMOD and DeepPrior++ on NYU.The distribution mass lies over the main diagonal; improvements are often very large on NYU, while LINEMOD gains remain possible where BB8 already performs well.Source: Figure 8 · p. 11, §6 · p. 11
  • Figure 8 compares per-image 2D Projection errors in pixels on LINEMOD and mean Euclidean distances in millimeters on NYU.The approach is on the x-axis and BB8 or DeepPrior++ on the y-axis, respectively.Source: Figure 8 · p. 11

7. Qualitative Results

Qualitative results compare object pose estimates with BB8 and hand pose estimates with DeepPrior++. The examples also indicate that LINEMOD estimates are accurate enough for augmented-reality applications.

7. Qualitative Results

  • On LINEMOD, the approach improves on BB8 both when BB8 fails for lack of matching training poses and when similar poses are present.Figure 9 illustrates cases where BB8 fails mostly because corresponding poses are absent; Figure 10 shows that accuracy can still often improve when similar poses are present.Source: §7 · p. 11, Figure 9 · p. 12, Figure 10 · p. 13
  • The LINEMOD pose estimates are reported as accurate enough for augmented-reality applications.Figure 11 renders “CVPR 2018” on a LINEMOD Can using an estimated pose.Source: §7 · p. 11, Figure 11 · p. 13
  • The NYU examples compare hand poses from the approach with estimates from DeepPrior++, using ground truth and predictions in contrasting colors.The top row shows DeepPrior++ estimates and the bottom row shows the approach; green marks ground truth and blue marks predictions.Source: §7 · p. 11, Figure 12 · p. 14, Figure 13 · p. 14
Loading 1712.03904v2…