PaperlayerView exact revision on arXiv

Source-linked AI summary

SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimation

Jiehong Lin, Lihua Liu, Dekun Lu, Kui Jia

arXiv:2311.15707v2cs.CV

TL;DR

Zero-shot 6D object pose estimation seeks to detect novel objects and estimate their poses in cluttered scenes. SAM-6D addresses this with SAM-based instance segmentation and two-stage point matching, outperforming existing methods on seven BOP datasets for both tasks.

  • Problem

    Zero-shot 6D object pose estimation aims to detect novel objects and estimate their 6D poses in cluttered scenes, beyond object-specific and category-limited settings.

  • Method

    SAM-6D uses ISM to generate and score proposals by semantics, appearance, and geometry, then PEM estimates poses through two-stage partial-to-partial point matching with background tokens.

  • Results

    SAM-6D significantly outperforms existing methods on seven core BOP datasets for both novel-object instance segmentation and pose estimation.

  • Takeaways & Limitations

    SAM-6D demonstrates robust generalization to novel-object instance segmentation and pose estimation across the evaluated BOP benchmark datasets.

  • Takeaways & Limitations

    The geometric matching confidence sgeo is easily impacted by occlusions, requiring a visible-ratio measure to assess its confidence.

Abstract

from arXiv · show

Zero-shot 6D object pose estimation involves the detection of novel objects with their 6D poses in cluttered scenes, presenting significant challenges for model generalizability. Fortunately, the recent Segment Anything Model (SAM) has showcased remarkable zero-shot transfer performance, which provides a promising solution to tackle this task. Motivated by this, we introduce SAM-6D, a novel framework designed to realize the task through two steps, including instance segmentation and pose estimation. Given the target objects, SAM-6D employs two dedicated sub-networks, namely Instance Segmentation Model (ISM) and Pose Estimation Model (PEM), to perform these steps on cluttered RGB-D images. ISM takes SAM as an advanced starting point to generate all possible object proposals and selectively preserves valid ones through meticulously crafted object matching scores in terms of semantics, appearance and geometry. By treating pose estimation as a partial-to-partial point matching problem, PEM performs a two-stage point matching process featuring a novel design of background tokens to construct dense 3D-3D correspondence, ultimately yielding the pose estimates. Without bells and whistles, SAM-6D outperforms the existing methods on the seven core datasets of the BOP Benchmark for both instance segmentation and pose estimation of novel objects.

1. Introduction

SAM-6D addresses zero-shot 6D pose estimation of novel objects in cluttered RGB-D scenes by combining SAM-based instance proposal selection with two-stage pose estimation. Its ISM uses semantic, appearance, and geometric matching, while PEM establishes dense 3D-3D correspondences through partial-to-partial point matching.

  • Motivation: Zero-shot 6D pose estimation seeks to detect novel object instances and estimate their 6D poses without object-specific training images.The task extends beyond instance-level methods requiring annotated target-object images and category-level methods restricted to known categories.
  • Framework: SAM-6D uses SAM to generate class-agnostic proposals, then retains valid novel-object proposals with matching scores based on semantics, appearance, and geometry.The geometric term uses shape and size information through bounding-box IoU with a rough pose projection.
  • Framework: PEM treats pose estimation as partial-to-partial point matching to handle occlusions, segmentation inaccuracies, and sensor noise.Its two-stage coarse-to-fine process establishes dense 3D-3D correspondence for pose calculation.
  • Training and evaluation: ISM requires no network retraining or fine-tuning, whereas PEM is trained on synthetic ShapeNet-Objects and Google-Scanned-Objects images.This separates SAM-based proposal generation from learned pose estimation.
  • Results: SAM-6D outperforms existing methods for both novel-object instance segmentation and pose estimation across the seven core BOP benchmark datasets.The framework evaluates LM-O, T-LESS, TUD-L, IC-BIN, ITODD, HB, and YCB-V.

2. Related Work

Related work spans promptable and semantically guided segmentation, efficient SAM variants, and image-matching approaches for estimating novel-object poses. These methods provide the segmentation and template-matching context in which SAM-6D is positioned.

  • Segment Anything: SAM defines a promptable segmentation task that predicts masks from points, boxes, text, or masks.Its model comprises an image encoder, prompt encoder, and mask decoder, with reported zero-shot transfer across diverse vision applications.
  • Semantic segmentation: Semantic segmentation methods extend SAM by assigning categories to masks or using reference images to locate category-specific objects.SSA adds semantic categories, while PerSAM and Matcher search for point prompts using same-category reference images.
  • Efficient variants: FastSAM and MobileSAM reduce SAM’s computational burden through convolutional networks or lightweight encoders.FastSAM replaces visual transformers with regular convolutional instance-segmentation networks, while MobileSAM uses decoupled distillation.
  • Image matching: Image-matching pose methods compare object proposals with rendered templates to retrieve viewpoint rotations and estimate final object poses.Gen6D, OVE6D, and GigaPose use image matching for viewpoint selection, while MegaPose employs a coarse estimator.

3. Methodology of SAM-6D

SAM-6D decomposes zero-shot 6D pose estimation into instance segmentation and pose estimation for novel objects in RGB-D scenes. ISM filters SAM-generated proposals using semantic, appearance, and geometric matching, while PEM estimates poses through staged partial point matching.

  • Instance Segmentation Model: ISM uses SAM to generate class-agnostic proposals and identifies instances matching a novel object.Proposals are filtered using confidence scores and non-maximum suppression before object matching.
  • Instance Segmentation Model: Each proposal receives a matching score combining semantic, appearance, and geometric terms.Templates are rendered from sampled SE(3) poses; semantic and patch-level appearance comparisons support the matching process.
  • Instance Segmentation Model: Geometric matching projects a coarse object pose into the image and computes bounding-box IoU, with visible ratio weighting geometric reliability under occlusion.The geometric score compares the proposal box with the projected object box.
  • Pose Estimation Model: PEM formulates pose estimation as partial-to-partial correspondence between sampled proposal and object point sets.The formulation accounts for occlusions, segmentation inaccuracies, and sensor noise.
  • Pose Estimation Model: Learnable background tokens represent non-overlapped points, enabling correspondence probabilities and weighted-SVD pose estimation.The attention matrix is converted into a soft assignment matrix, whose maxima identify point matches or background assignments.
  • Pose Estimation Model: PEM performs coarse and fine point matching, using sparse correspondences for an initial pose and dense correspondences for the final pose.Sparse-to-Dense Point Transformers support dense feature interactions in the fine matching stage.

4. Experiments

SAM-6D is evaluated for novel-object instance segmentation and pose estimation across the seven core BOP datasets, with ablations examining its matching designs and runtime. The experiments report improvements over existing methods, faster background-token inference than optimal transport, and efficiency without render-based refinement.

  • Evaluation setup: SAM-6D is evaluated on seven core BOP datasets for instance segmentation and pose estimation of novel objects.The reported metrics are mAP for segmentation and mean AR across VSD, MSSD, and MSPD for pose estimation.
  • Instance segmentation: Its ISM outperforms semantic-only baselines, with further gains from adding appearance and geometry matching scores.The comparisons include ZeroPose and CNOS, while the semantic-only SAM/FastSAM baseline aligns with CNOS.
  • Pose estimation: Its PEM outperforms MegaPose, ZeroPose, and GigaPose under various mask predictions without the time-intensive render-based refiner.Using masks predicted by ISM further improves PEM performance relative to other mask predictions.
  • Ablation studies: Background tokens achieve results comparable to optimal transport while providing faster inference for partial-to-partial point matching.Optimal transport becomes more time-consuming as the density of matched points increases.
  • Ablation studies: The fine point-matching stage improves coarse matching results, while runtime varies with the segmentation model because SAM generates more proposals than FastSAM.Runtime is reported as average per-image processing time on a GeForce RTX 3090 GPU.

5. Conclusion

SAM-6D addresses zero-shot 6D pose estimation through two sequential models: ISM proposes and scores candidate objects, while PEM estimates poses through staged point matching. Across seven BOP datasets, the framework significantly outperforms existing methods.

  • Framework: SAM-6D combines ISM and PEM to perform novel-object instance segmentation and pose estimation from RGB-D images.ISM scores proposals using semantics, appearance, and geometry; PEM uses coarse and fine point matching.
  • Conclusion: SAM-6D significantly outperforms existing methods on the seven core BOP benchmark datasets.

Supplementary Material

The supplementary material provides additional details and results for the instance segmentation and pose estimation models.

  • Instance Segmentation: The supplementary material includes additional instance-segmentation analyses, quantitative results, model-size effects, and qualitative comparisons.
  • Pose Estimation: It also provides pose-estimation architecture, feature extraction, matching, training-objective, quantitative, and qualitative-comparison details.
  • Organization: The supplementary material is organized as an appendix to the paper.

Instance Segmentation Model

The ISM uses template-based patch matching to estimate visibility and weight geometric matching reliability. It constructs target templates from fully visible viewpoints and uses specified segmentation and description models.

  • Visible ratio: The visible ratio rvis weights the reliability of the geometric matching score sgeo.It is computed from correspondences between patches in a proposal crop and the best-matched object template.
  • Visible ratio: rvis estimates the object's occlusion degree by counting template patches that find corresponding patches in the proposal crop.The ratio is based on the number of matched template patches relative to the total template patches.
  • Visible ratio: The visibility indicator assigns 0 when svis,i < δvis and 1 when svis,i ≥ δvis.The threshold δvis is empirically set to 0.5.
  • Template construction: The target object is represented by 42 fully visible templates sampled from well-distributed icosphere viewpoints.Templates are cropped and background-masked from BOP physically based rendering training images.
  • Model configuration: The implementation uses SAM or FastSAM for segmentation and ViT-L of DINOv2 as the description model.SAM is based on ViT-H, while FastSAM is based on YOLOv8x.

A.4.1 Detection Results

The detection evaluation compares ISM with existing methods and examines model size and matching-score components. Qualitative results illustrate the effects of appearance and geometric matching scores.

  • Model size: Larger segmentation and description models correlate with higher performance on YCB-V.The comparison uses mAP scores across multiple IoU thresholds.
  • Matching-score analysis: The appearance matching score is qualitatively compared with and without its use for proposal scoring.The comparison is shown in Figure 4.
  • Matching-score analysis: The geometric matching score is qualitatively compared with and without its use for proposal scoring.The comparison is shown in Figure 5.

A.5.3 More Qualitative Comparisons with Existing Methods

The pose-estimation pipeline extracts point features, performs coarse and fine matching with background tokens, and estimates poses from weighted correspondences. Qualitative and tabulated materials cover BOP datasets, model settings, runtime, and template-view choices.

  • Qualitative comparisons: Qualitative comparisons cover novel-object instance segmentation across all seven core BOP datasets.Ground-truth masks are provided except for ITODD and HB, where they are unavailable.
  • Feature Extraction: The Feature Extraction module uses ViT-Base to produce per-pixel features from masked RGB crops of proposals and rendered templates.Observed-image pixels inside the mask correspond to points transformed from masked depth, while template pixels correspond to rendered surface points.
  • Evaluation materials: The reported materials include detection metrics, model-size comparisons, runtime comparisons, and template-view evaluations.Template-view results use mean Average Recall among VSD, MSSD, and MSPD.
  • Feature Extraction: Both proposal and target point sets are normalized to a unit sphere to address object-scale variation.The normalization divides coordinates by object scale.
  • Coarse Point Matching: The coarse matching module uses Geometric Transformers and background tokens to compute point correspondence probabilities.The probabilities are sharpened, normalized, and used to sample point-pair triplets for pose hypotheses.
  • Coarse Point Matching: The coarse stage generates 6,000 pose hypotheses and retains the 300 with the smallest point-pair distances.The best pose among the retained hypotheses initializes fine point matching.
  • Fine Point Matching: Fine matching uses positional encodings and Sparse-to-Dense Point Transformers to model dense relationships.Geometric Transformers process sparse point sets, while Linear Cross-attention disseminates information to dense points.
  • Fine Point Matching: The final pose is calculated by weighted SVD after assigning each proposal point its best-matched target point.Matching probabilities serve as the weights.

B.2. Training Objectives

Training supervises coarse and fine attention matrices using cross-entropy targets derived from pose-aligned point correspondences. Template-view experiments and an OVE6D comparison provide additional evaluation context.

  • Attention supervision: The attention matrix is supervised with cross-entropy losses for proposal-to-object and object-to-proposal assignments.The objective applies CE to both directional correspondence targets.
  • Correspondence targets: Ground-truth correspondence labels select the closest pose-aligned target point when its distance is below δdis.The distance threshold δdis is set to 0.15 because point sets are normalized to a unit sphere.
  • Optimization: The training objective sums losses across all transformer blocks in both coarse and fine point matching modules.Upper scripts distinguish coarse and fine losses, while the lower block index identifies transformer depth.
  • Template-view evaluation: Using more object-template views improves pose-estimation performance, while one view performs poorly because it cannot fully depict the object.The default setting uses 42 views for consistency with ISM and CNOS.
  • Comparison with OVE6D: SAM-6D outperforms OVE6D on LM-O using the same segmentation masks and without ICP post-optimization.OVE6D retrieves viewpoint rotations from template image matching and regresses in-plane rotation.

B.4. More Qualitative Comparisons with Existing Methods

Figure 10 presents qualitative pose-estimation comparisons for novel objects across all seven core BOP datasets, including comparisons with MegaPose and available ground truths.

  • Corresponding ground-truth poses are shown for comparison except on ITODD and HB, where they are unavailable.
Loading 2311.15707v2…