PaperlayerView exact revision on arXiv

Source-linked AI summary

FS6D: Few-Shot 6D Pose Estimation of Novel Objects

Yisheng He, Yao Wang, Haoqiang Fan, Jian Sun, Qifeng Chen

arXiv:2203.14628v1cs.CVcs.RO

TL;DR

Few-shot 6D pose estimation addresses the restriction of close-set methods to known objects and precise CAD models. The paper proposes dense RGBD prototype matching and ShapeNet6D for prior learning, and reports outperforming local image feature matching and point-cloud registration methods on LineMOD and YCB-Video.

  • Problem

    Close-set pose-estimation methods assume known object instances or predefined categories and often rely on high-fidelity CAD models, limiting evaluation on novel objects.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    The paper formulates an open-set task using a few labeled RGBD support views and proposes FS6D-DPM to match dense prototypes using appearance and geometry.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    On LineMOD and YCB-Video, the method outperforms state-of-the-art local image feature matching and point-cloud registration methods by large margins.

    Source: §4.5 · p. 8

  • Takeaways & Limitations

    The authors establish an open-set few-shot 6D pose estimation problem and benchmark to support its study.

    Source: §1 · p. 2

    Few-shot algorithms still have a significant performance gap from methods trained under the close-set setting.

    Source: §5 · p. 8

Abstract

from arXiv · show

6D object pose estimation networks are limited in their capability to scale to large numbers of object instances due to the close-set assumption and their reliance on high-fidelity object CAD models. In this work, we study a new open set problem; the few-shot 6D object poses estimation: estimating the 6D pose of an unknown object by a few support views without extra training. To tackle the problem, we point out the importance of fully exploring the appearance and geometric relationship between the given support views and query scene patches and propose a dense prototypes matching framework by extracting and matching dense RGBD prototypes with transformers. Moreover, we show that the priors from diverse appearances and shapes are crucial to the generalization capability under the problem setting and thus propose a large-scale RGBD photorealistic dataset (ShapeNet6D) for network pre-training. A simple and effective online texture blending approach is also introduced to eliminate the domain gap from the synthesis dataset, which enriches appearance diversity at a low cost. Finally, we discuss possible solutions to this problem and establish benchmarks on popular datasets to facilitate future research. The project page is at \url{https://fs6d.github.io/}.

1. Introduction

The paper frames few-shot 6D pose estimation as an open-set alternative to methods restricted to known objects or predefined categories. It proposes a dense RGBD matching framework, a dataset for prior learning, and a benchmark for the task.

1. Introduction

  • Few-shot 6D pose estimation targets novel objects from a few labeled views without extra training or precise CAD models.This open-set task differs from close-set methods that assume training and testing objects come from the same object space.Source: §1 · p. 1
  • ShapeNet6D provides diverse shapes and appearances for prior learning, alongside online texture blending to enrich appearance at low cost.The authors report a dataset of 800K images and 12K objects.Source: §1 · p. 2
  • The authors introduce the open-set few-shot pose estimation problem and establish a benchmark to study it.They position the task as relevant to applications such as fast registration of novel objects for robotic manipulation.Source: §1 · p. 2
  • FS6D-DPM matches dense RGBD prototypes to leverage appearance and geometric information for pose estimation.The framework is motivated by the need to explore relationships between support views and the query scene patch.Source: §1 · p. 2

2. Related Work

Prior pose-estimation methods commonly assume known instances or predefined categories. The paper contrasts these with few-shot approaches, whose pose task demands dense correspondence and complementary appearance and geometry cues.

2.1. 6D Object Pose Estimation in Close-Set Setting

  • Instance-level methods assume known objects and precise CAD models, while category-level methods generalize only to unseen instances within predefined categories.The paper's open-set setting instead targets unknown objects without extra training, whether or not they belong to trained categories.Source: §2.1 · p. 2

2.2. Possible Few-Shot Pose Estimation Solutions

  • Local image matching uses grayscale cues and drops in performance on textureless objects; point-cloud methods rely on fine depth data and fail on reflective objects.The paper motivates using complementary RGBD information for pose estimation.Source: §2.2 · p. 2

2.3. Metric learning on few-shot learning problems

  • Unlike classification and segmentation methods that can use sparse or pooled prototypes, few-shot pose estimation requires dense pixel-level correspondences between support prototypes and query features.The cited metric-learning approaches use global or part-level prototypes for their respective tasks.Source: §2.3 · p. 2

3. Proposed Method

The paper defines few-shot 6D pose estimation as an open-set task using a few labeled support views to estimate a novel object's pose in a query scene. It also identifies domain generalization from synthetic data to real scenes as a related challenge.

3.1. Problem Formulation

  • Given a few labeled RGBD patches of a novel object, the task is to retrieve its 6D pose in a novel query scene without extra training.The formulation does not require precise CAD models.Source: §3.1 · p. 3
  • The open-set formulation removes the requirement that training and testing objects belong to the same predefined instance or category spaces.It is intended to apply to novel objects whether or not they fall within trained categories.Source: §3.1 · p. 3
  • Domain generalization aims to reduce gaps between synthetic and real-world data, but large-scale real-world data collection and labeling is costly.The paper identifies leveraging diverse synthetic shape and appearance priors for real-world generalization as an important research problem.Source: §3.1 · p. 3

3.2. Datasets

The paper introduces ShapeNet6D to provide diverse shape and appearance priors for few-shot pose estimation, and describes online texture blending as a low-cost way to enrich rendered objects with real-image appearance.

3.2. Datasets

  • Shape diversity and semantic priors are considered important for generalization, while existing 6D pose datasets are described as small and insufficiently diverse.The authors propose ShapeNet6D as a large-scale dataset for prior learning.Source: §3.2 · p. 3

3.2.1 ShapeNet6D

  • ShapeNet6D contains photorealistic RGBD scenes of more than 12K ShapeNet objects, with instance segmentation and pose labels.Scenes are rendered with randomized objects, materials, textures, backgrounds, and lighting; the authors report that its greater shape and appearance diversity provides better priors empirically.Source: §3.2.1 · pp. 3–4

3.2.2 Online texture blending

  • Online texture blending maps real-world image colors onto object meshes without artificial simulation, preserving the real-image domain while enriching appearance diversity at low cost.The procedure uses UV coordinates to map image texture onto mesh pixels during rasterization.Source: Figure 2 · p. 4, §3.2.2 · p. 4

3.3. FS6D-DPM

FS6D-DPM estimates novel-object poses by matching dense RGBD support prototypes to query-scene features. Its pipeline combines Siamese feature extraction, transformer attention for correspondence, and geometric pose recovery.

3.3.1 Preliminaries

  • Dense RGBD prototypes establish local-to-local support/query correspondences, unlike global-to-global or global-to-local matching used in few-shot classification and segmentation.RGB texture is a cue for smooth-surface objects, while depth geometry is important for textureless objects.Source: §3.3.1 · p. 4, Figure 4 · p. 5

3.3.2 Overview

  • The overview extracts support and query features, applies attention to obtain dense prototypes and query features for correspondence, then recovers pose parameters.The framework uses a Siamese RGBD backbone and transformer-based dense prototype extraction before pose estimation.Source: §3.3.2 · p. 5, Figure 3 · p. 5

3.3.3 Feature Extraction Backbone

  • A Siamese FFB6D backbone extracts semantic and geometric features from support images and query scenes.Bidirectional local feature fusion blocks in FFB6D bridge information across encoding and decoding layers.Source: §3.3.3 · p. 5

3.3.4 Dense Prototypes Extraction and Matching

  • Linear Transformers use self-attention to enhance support and query features and cross-attention to exchange context before inner-product similarity and Sinkhorn matching.Scene features query support prototypes, and the reverse direction is also used to enhance contextual information.Source: §3.3.4 · pp. 5–6

3.3.5 Pose Parameters Estimation

  • Umeyama estimates rotation and translation from matched 3D coordinates; RANSAC addresses outliers, and the lowest-loss prediction is selected across K support views.The pose parameters minimize the squared alignment error between matched prototype and query coordinates.Source: §3.3.5 · p. 6

4. Experiments

The experiments benchmark few-shot 6D pose estimation on LineMOD and YCB-Video, comparing FS6D-DPM with local-feature, point-cloud, and template-matching baselines. They report the proposed method’s benchmark results and examine ShapeNet6D pre-training and online texture blending.

4.1. Benchmark Datasets

  • The benchmarks use 16 support shots per object and rotate three object groups through training and testing splits on each dataset.LineMOD contains 13 videos of 13 low-textured objects; YCB-Video contains 92 RGBD videos of 21 objects.Source: §4.1 · p. 6

4.2. Evaluation Metrics

  • YCB-Video reports ADD and ADDS area-under-curve scores, while LineMOD reports recall at an error below 10% of object diameter.ADD evaluates asymmetric objects; ADDS uses closest-point distance for symmetric objects.Source: §4.2 · p. 6

4.3. Baselines

  • The baselines represent local image feature matching, point-cloud registration, and template matching; template matching’s reported upper bound uses a closest-rotation view and center-shift translation.All methods are compared without iterative refinement such as ICP.Source: §4.3 · pp. 6, 8

4.5. Benchmark Results

  • FS6D-DPM achieves the best performance among the evaluated few-shot pose baselines on both YCB-Video and LineMOD.Table 2 covers YCB-Video and Table 3 covers LineMOD; Figure 5 shows qualitative results alongside ground truths.Source: Table 2 · p. 7, Table 3 · p. 7, Figure 5 · p. 7

4.4. Training and Implementation

  • The models are pre-trained on ShapeNet6D with online augmentation, fine-tuned on benchmark datasets, and given 16 support views per object.The reported setup uses two pre-training epochs and five fine-tuning epochs.Source: §4.4 · p. 8

4.6. Ablation Study

  • On LineMOD, ShapeNet6D pre-training improves performance by 11% over training from scratch, while the pre-trained model is 4.1% behind the fine-tuned one.The authors attribute the remaining domain-generalization drop mainly to domain gap; online texture blending improves performance on texture-rich YCB-Video objects by large margins.Source: §4.6 · p. 8, §4.5 · p. 8, Table 5 · p. 8

5. Discussion and Limitations

The discussion identifies limits in the pose-estimation pipeline, synthetic-to-real generalization, and the performance gap to close-set methods.

5. Discussion and Limitations

  • The approach relies on object detectors to crop regions of interest rather than jointly detecting objects and estimating pose.The authors describe a joint framework as more practical.Source: §5 · p. 8
  • ShapeNet6D is synthetic, and the domain-gap problem remains unresolved despite its shape and appearance diversity.Suggested directions include domain-invariant pose estimation or large-scale real-world datasets.Source: §5 · p. 8
  • Few-shot methods still have a significant performance gap from methods trained under the close-set setting.The authors point to 3D keypoint-based techniques as one possible direction for narrowing it.Source: §5 · p. 8

A.1. More Results

The LineMOD comparison reports that the model surpasses iteratively refined LatentFusion without refinement and runs substantially faster under the same support-view setting. More support views improve performance, with only marginal gains beyond 16 views.

Model running time

  • Each support view requires an average of 72 ms for neural network forwarding and 113 ms for Umeyama pose alignment with RANSAC on an RTX 2080Ti.Source: §A.1 · p. 9

Pose estimation with ground-truth segmentation

  • 0.34 fps versus 0.018 fps: the model is 18X faster than LatentFusion on an RTX 2080Ti with 16 support views, and without refinement it surpasses iteratively refined LatentFusion.The model is trained only with ShapeNet6D and not fine-tuned on real LineMOD data; ICP post-refinement improves its results further.Source: §A.1 · p. 9

Effect of the different number of support views

  • Performance improves as support views increase, but gains are only marginal beyond 16 views.The authors report that good pose results are achievable with few support views.Source: §A.1 · p. 9

A.2. Implementation Details

The implementation splits LineMOD and YCB-Video objects into non-overlapping groups for novel-object testing. Support views are selected from training data using farthest rotation sampling, then used to define an object-coordinate reference frame.

Grouping information of benchmark datasets

  • Both LineMOD and YCB-Video are split into three non-overlapping groups, with two groups used for training and the third as novel-object test data.Source: §A.2 · p. 9

Support views selection

  • For each object, 16 training-set views are selected by farthest rotation sampling, initialized randomly and iteratively expanded to maximize rotation distance and visibility of object parts.Training-set views are used because real-world objects in these datasets are unavailable for support-view capture.Source: §A.2 · p. 9
  • Rotation distance is the smaller Euclidean distance between unit quaternions q1 and q2, considering both ||q1 - q2|| and ||q1 + q2||.Source: §A.2 · p. 9
  • The selected support-view masks and poses define a reference frame by transforming the cropped object point cloud back into object coordinates.Source: §A.2 · p. 9

A.3. Fast Registration of Novel Objects

The paper describes two ways to obtain support views for novel objects: selecting them from an RGBD video or capturing them at predefined robot-arm viewpoints. Video-derived relative poses can be improved with a marker board.

Select from an RGBD video of the novel object

  • An RGBD video can supply support views by defining coordinates from its first frame, registering later frames to it, and selecting views by farthest rotation sampling.A clean plane helps remove the background and register adjacent frames; a marker board can improve relative-pose accuracy.Source: §A.3 · p. 10

Collect with the robot arm

  • A calibrated robot arm can capture support views at predefined camera viewpoints, with pose parameters supplied by the robot’s known poses.Source: §A.3 · p. 10
Loading 2203.14628v1…