PaperlayerView exact revision on arXiv

Source-linked AI summary

EfficientPose: An efficient, accurate and scalable end-to-end 6D multi object pose estimation approach

Yannick Bukschat, Marcus Vetter

arXiv:2011.04307v2cs.CV

TL;DR

Methods that first detect 2D targets and then solve PnP for each object can incur increasing runtime as object counts rise. EfficientPose extends EfficientDet for single-shot multi-object 6D pose estimation and introduces 6D augmentation. It reports a new state-of-the-art Linemod result while running end-to-end at over 27 FPS.

  • Problem

    RGB pose-estimation pipelines that detect 2D targets and solve PnP per object have runtime that increases with object count, a concern for applications with runtime limitations.

    Source: §1 · p. 2

  • Method

    EfficientPose extends EfficientDet with translation and rotation prediction subnetworks for single-shot multi-object 6D pose estimation and proposes 6D augmentation.

    Source: §1 · p. 2

  • Results

    Over 27 FPS, EfficientPose achieves a new state-of-the-art result on Linemod while running end-to-end.

    Source: §5 · p. 12

  • Takeaways & Limitations

    Its direct single-shot pose prediction makes runtime nearly independent of object count while handling multiple categories and instances.

    Source: §1 · p. 2

    Scaling augmentation introduces error when the object is not at the image center, because image rescaling does not reproduce the projection change from moving the object along the z-axis.

    Source: §3.5 · p. 7

Abstract

from arXiv · show

In this paper we introduce EfficientPose, a new approach for 6D object pose estimation. Our method is highly accurate, efficient and scalable over a wide range of computational resources. Moreover, it can detect the 2D bounding box of multiple objects and instances as well as estimate their full 6D poses in a single shot. This eliminates the significant increase in runtime when dealing with multiple objects other approaches suffer from. These approaches aim to first detect 2D targets, e.g. keypoints, and solve a Perspective-n-Point problem for their 6D pose for each object afterwards. We also propose a novel augmentation method for direct 6D pose estimation approaches to improve performance and generalization, called 6D augmentation. Our approach achieves a new state-of-the-art accuracy of 97.35% in terms of the ADD(-S) metric on the widely-used 6D pose estimation benchmark dataset Linemod using RGB input, while still running end-to-end at over 27 FPS. Through the inherent handling of multiple objects and instances and the fused single shot 2D object detection as well as 6D pose estimation, our approach runs even with multiple objects (eight) end-to-end at over 26 FPS, making it highly attractive to many real world scenarios. Code will be made publicly available at https://github.com/ybkscht/EfficientPose.

1. Introduction

For applications that need 6D poses, 2D-target-and-PnP pipelines can become costly with multiple objects because pose solving is repeated per object. EfficientPose extends EfficientDet for single-shot multi-object pose estimation, and adds 6D augmentation to improve performance and generalization.

1. Introduction

  • Over 26 FPS, Figure 1 shows φ = 0 making single-shot 6D multi-object pose predictions on the Occlusion test set.The figure also plots average end-to-end FPS for φ = 0 and φ = 3 against objects per image.Source: Figure 1 · p. 1
  • 2D-target-and-PnP approaches compute each object’s pose individually, so runtime increases linearly with object count and can limit use in multi-object, time-constrained scenarios.Some approaches also apply RANSAC voting or a separate detector and crop to each object.Source: §1 · p. 2
  • EfficientPose adds translation and rotation subnetworks to EfficientDet, sharing input feature-map computation to predict multiple objects’ 6D poses in a single shot with nearly object-count-independent runtime.The direct pose regression requires no RANSAC or PnP post-processing.Source: §1 · p. 2
  • 6D augmentation is identified as a key element of the reported Linemod ADD(-S) accuracy and is proposed to improve performance and generalization, especially on small datasets.Image rotation and scaling would otherwise mismatch the image and annotated poses.Source: §1 · p. 2
  • EfficientPose extends EfficientDet to retain single-shot multi-object and instance detection while adding 6D pose estimation, scalability, efficiency, and ease of use.The approach uses a single hyperparameter to scale across computational resources, speed, and accuracy.Source: §1 · p. 2

2. Related Work

Prior work on RGB 6D pose estimation includes direct pose estimation and pipelines that detect 2D targets before solving PnP; this section also reviews the 2D detectors underlying EfficientPose.

2.1. Direct estimation of the 6D pose

  • Direct pose estimation includes regression of pose components and discretized rotation classification; discretized poses may be coarse and require further refinement.PoseCNN decouples translation and rotation and uses ShapeMatch-Loss for symmetric objects; the paper bases its own loss on that function.Source: §2.1 · pp. 2–3

2.2. 2D Detection and PnP

  • 2D detection-and-PnP methods use keypoints or dense 2D–3D correspondences, but per-object pose solving makes runtime object-count-dependent and can complicate multi-object or instance handling.Some approaches also require an external detector, separate object models, or additional modifications.Source: §2.2 · p. 3

2.3. 2D Object Detection

  • EfficientDet scales a one-stage detector with a single hyperparameter across computational resources, motivating its use as EfficientPose’s base architecture.One-stage detectors perform detection in a single shot; two-stage approaches first generate region proposals.Source: §2.3 · p. 3

3. Methods

EfficientPose extends a 2D detector to predict object rotation and translation from RGB images. Its added pose subnetworks retain single-shot handling of multiple object instances and categories.

3.1. Extending the EfficientDet architecture

  • EfficientPose adds rotation and translation subnetworks to EfficientDet, sharing feature maps with its classification and box heads for single-shot multi-object pose prediction.The pose subnetworks add minimal computational cost, and their size is controlled by the EfficientDet scaling parameter φ.Source: §3.1 · p. 4

3.2. Rotation Network

  • Rotation is represented by axis-angle vectors and iteratively refined by predicting Δr from the current rotation and convolutional features.Group normalization allows training the rotation and translation networks from scratch with batch size 1.Source: §3.2 · pp. 4–5

3.3. Translation Network

  • Translation prediction separates the 2D object center from depth, then recovers 3D translation using camera intrinsics under a pinhole-camera assumption.Center offsets are predicted per anchor and normalized by feature-map stride; translation recovery and center-coordinate calculations are implemented in separate layers.Source: §3.3 · pp. 5–6

3.4. Transformation Loss

  • The transformation loss compares ground-truth and estimated transformed model points, using nearest-point distances for symmetric objects to avoid unnecessary training penalties.For asymmetric objects, mean point distance corresponds to ADD and directly optimizes the evaluation metric without balancing separate rotation and translation losses.Source: §3.4 · pp. 6–7

3.5. 6D Augmentation

  • 6D augmentation rotates images and adjusts pose labels together, while image scaling adjusts translation depth; scaling introduces error when objects are off-center.Experiments sample rotation uniformly from [0°, 360°) and scaling from [0.7, 1.3].Source: §3.5 · pp. 7–8

3.6. Color space Augmentation

  • Color-space augmentation uses RandAugment transformations and Gaussian noise while excluding rotations and shearing that would mismatch the image and pose labels.The augmentation strength is controlled by the number and strength of transformations.Source: §3.6 · p. 8

4. Experiments

The experiments section presents the experimental setup, implementation details, and evaluation metrics, and compares Linemod results with state-of-the-art methods. Although the approach can scale from φ = 0 to φ = 7, experiments use only φ = 0 and φ = 3 because of computational constraints.

4. Experiments

  • Experiments compare the approach with state-of-the-art methods on Linemod.The section also describes its setup, implementation details, and evaluation metrics.Source: §4 · p. 8
  • φ can range from 0 to 7 in integer steps, but experiments use only φ = 0 and φ = 3 because of computational constraints.Source: §4 · p. 8

4.1. Datasets

The study evaluates on Linemod and Occlusion. Linemod experiments follow a per-object training protocol, while Occlusion is used to assess multi-object pose estimation under partial occlusion and a distinct, non-comparable evaluation protocol.

4.1.1 Linemod

  • Linemod has 13 commonly used objects in 13 cluttered scenes, but only one object per scene is annotated with a 6D pose.Source: §4.1.1 · p. 8
  • Because only one object per Linemod scene is annotated, the study trains a separate model for each object despite the approach supporting multiple objects.Source: §4.1.1 · p. 8
  • Linemod uses about 1,200 annotated examples per object and the prior-work split, with roughly 15% training images, a minimum 15° angular distance between training poses, and no synthetic training images.Source: §4.1.1 · p. 8

4.1.2 Occlusion

  • Occlusion evaluates multi-object pose estimation using one model trained on a scene where eight other objects are additionally annotated, including heavily occluded objects.Its training and test split matches the corresponding Linemod scene, but its results are not comparable with works that train on Linemod and test on all Occlusion data.Source: §4.1.2 · p. 8

4.2. Evaluation metrics

The evaluation uses ADD(-S), based on average distances between model points transformed by ground-truth and estimated poses. It distinguishes asymmetric and symmetric objects, with correctness determined by a threshold relative to object diameter.

4.2. Evaluation metrics

  • ADD(-S) compares the average distances between 3D model points transformed by ground-truth and estimated poses.The metric uses ADD for asymmetric objects and ADD-S for symmetric objects.Source: §4.2 · pp. 8–9
  • An estimated pose is correct when its average point distance is below 10% of the object's diameter.Source: §4.2 · p. 8
  • Table 1 compares methods on Linemod using ADD(-S), marking symmetric objects with * and methods using additional refinement with +.Source: Table 1 · p. 9

4.3. Implementation Details

Training uses Adam with a small batch size and a long schedule, and combines classification, box-regression, and transformation losses. The implementation uses the paper's 6D and color-space augmentations and COCO-pretrained EfficientDet weights for most of the network.

4.3. Implementation Details

  • Training uses Adam at an initial learning rate of 1e-4, batch size 1, gradient clipping at 0.001, and runs for 5,000 epochs.The learning rate halves if average point distance does not decrease over 25 test-set evaluations, with a minimum of 1e-7; evaluation occurs every 10 epochs.Source: §4.3 · p. 9
  • The training objective combines classification, bounding-box regression, and transformation losses as L = λclass · Lclass + λbbox · Lbbox + λtrans · Ltrans, with λclass = λbbox = 1 and λtrans = 0.02.The transformation loss uses m = 500 points from the 3D object model point set.Source: §4.3 · p. 9
  • The implementation applies 6D and color-space augmentation by default and initializes all but the rotation and translation networks with COCO-pretrained vanilla EfficientDet weights.Batch-normalization layers are frozen using population statistics learned from COCO because the batch size is small.Source: §4.3 · p. 9

4.4. Comparison on Linemod

On Linemod, EfficientPose competes with state-of-the-art RGB methods while estimating poses directly in a single shot, without PnP or further post-processing. The φ = 0 and φ = 3 models perform nearly the same on this single-object task.

4.4. Comparison on Linemod

  • EfficientPose outperforms all compared methods without refinement, including DPOD+, and roughly halves DPOD+’s remaining error.The authors report that this makes single-shot direct pose estimation competitive with methods such as PVNet, DPOD, and HybridPose.Source: §4.4 · pp. 9–10
  • Unlike the compared recent works, EfficientPose estimates object poses in one shot without PnP or further post-processing such as RANSAC-based voting.The authors report that even DPOD+ uses an additional refinement network.Source: §4.4 · pp. 9–10
  • Figure 8 uses green boxes for ground-truth poses and blue boxes for EfficientPose estimates.The figure shows qualitative predictions from the φ = 0 model on the Linemod test set.Source: Figure 8 · p. 10
  • The φ = 0 and φ = 3 models perform nearly the same despite their different capacities on single-object Linemod.The authors suggest that the smaller model may avoid some overfitting and that the small dataset, rather than model capacity, is the bottleneck.Source: §4.4 · p. 10

4.5. Multi object pose estimation

A single EfficientPose model detects objects and estimates their 6D poses in one shot on the Occlusion dataset. The authors caution that this experiment’s results are not comparable to prior work because it uses a different training and test setup.

4.5. Multi object pose estimation

  • A single model detects and estimates poses for multiple objects in one shot on the Occlusion dataset, but its results are not comparable to prior work because the data split differs.The experiment used the train and test split of the corresponding Linemod scene because the Linemod training data used by other works could not be used here.Source: §4.5 · p. 10, Table 2 · p. 10
  • Figure 9 shows qualitative Occlusion predictions, with ground-truth poses in green and estimated poses in other colors.The figure illustrates the single φ = 0 model’s multi-object pose-estimation ability.Source: Figure 9 · p. 10
  • The φ = 3 model performs substantially better than φ = 0 on Occlusion, unlike the nearly equal results on Linemod.The authors suggest that the larger number of objects favors greater model capacity; severe occlusions also make this task more challenging than Linemod.Source: §4.5 · p. 10

4.6. Runtime analysis

The runtime analysis measures preprocessing, network inference, and end-to-end time for single- and multi-object pose estimation. EfficientPose’s end-to-end runtime changes little with object count, while its extension of EfficientDet incurs a modest frame-rate reduction.

4.6. Runtime analysis

  • The analysis separately measures preprocessing, pure network inference, and end-to-end runtime, including nonmaximum-suppression and post-processing.It compares single-object Linemod and multi-object Occlusion scenarios at batch size 1.Source: §4.6 · pp. 10–11
  • 27.45 FPS for single-object Linemod and 26.22 FPS for multi-object Occlusion are the φ = 0 model’s average end-to-end rates.Occlusion typically contains eight annotated objects per image.Source: §4.6 · p. 11
  • 9.43 FPS for single-object and 9.34 FPS for multi-object estimation are the φ = 3 model’s end-to-end rates.The authors report that runtime is nearly independent of the number of objects per image.Source: §4.6 · p. 11
  • EfficientPose’s runtime is nearly independent of object count, unlike 2D detection plus PnP approaches whose runtime increases linearly with the number of objects.The authors report roughly 35 vs. 27 FPS for vanilla EfficientDet and EfficientPose at φ = 0, and 12 vs. 9 FPS at φ = 3.Source: §4.6 · p. 11

4.7. Ablation study

The ablation study tests 6D augmentation on the Linemod driller object and finds a large improvement in ADD. Scale-only and rotation-only variants show similar improvements.

4.7. Ablation study

  • 99.9% ADD with 6D augmentation versus 72.15% without it shows a large improvement on the driller object.The ablation used the φ = 0 model and applied color-space augmentation in all experiments.Source: §4.7 · p. 11, Table 4 · p. 12
  • The ablation is limited to the Linemod driller object because of computational constraints.The reported results use the φ = 0 model and ADD(-S) metric.Source: Table 4 · p. 12
  • Scale-only and rotation-only 6D augmentation produce very similar improvements, suggesting that both contribute equally to its effectiveness.The study evaluates the individual augmentation components on the Linemod driller object.Source: §4.7 · p. 12, Table 4 · p. 12

5. Conclusion

EfficientPose combines 2D detection and 6D pose estimation for multiple objects in one shot while retaining high accuracy and speed. Its runtime is nearly independent of object count, supporting real-time use in multi-object settings.

5. Conclusion

  • State-of-the-art Linemod performance with RGB input runs end-to-end at over 27 FPS.The conclusion reports this result for EfficientPose overall.Source: §5 · p. 12
  • EfficientPose extends EfficientDet with low-cost translation and rotation subnetworks for single-shot detection and 6D pose estimation of multiple objects and instances.The subnetworks share input computation with the base network, keeping additional computational costs low.Source: §5 · p. 12
  • Unlike 2D+PnP approaches, EfficientPose's runtime is nearly independent of object count, making it suitable for real-time multi-object scenarios such as robotic grasping and autonomous driving.The paper attributes this suitability to handling multiple objects in one shot.Source: §5 · p. 12

Related papers

Loading 2011.04307v2…