PaperlayerView exact revision on arXiv

Source-linked AI summary

Unseen Object Instance Segmentation for Robotic Environments

Christopher Xie, Yu Xiang, Arsalan Mousavian, Dieter Fox

arXiv:2007.08073v2cs.CVcs.RO

TL;DR

The paper tackles unseen object instance segmentation for tabletop robots despite limited real-world training data. UOIS-Net separates depth-based mask initialization from RGB refinement and trains on synthetic RGB-D data, achieving strong real-world segmentation performance and supporting unseen-object grasping.

  • Problem

    Robots need to segment unseen tabletop objects, but large-scale diverse annotated datasets for this task are expensive and generally unavailable.

  • Method

    UOIS-Net predicts rough instance masks from depth-based 2D or 3D center votes, refines them with RGB, and trains on the synthetic Tabletop Object Dataset.

  • Results

    UOIS-Net produces sharp, accurate masks, outperforms Mask R-CNN and PointGroup, and improves OCID overlap and boundary F-measures over Mask R-CNN by 8.1% and 6.0%, respectively.

  • Takeaways & Limitations

    The framework generalizes from non-photorealistic synthetic RGB-D data to real tabletop scenes and supports robotic interaction with unseen objects.

Abstract

from arXiv · show

In order to function in unstructured environments, robots need the ability to recognize unseen objects. We take a step in this direction by tackling the problem of segmenting unseen object instances in tabletop environments. However, the type of large-scale real-world dataset required for this task typically does not exist for most robotic settings, which motivates the use of synthetic data. Our proposed method, UOIS-Net, separately leverages synthetic RGB and synthetic depth for unseen object instance segmentation. UOIS-Net is comprised of two stages: first, it operates only on depth to produce object instance center votes in 2D or 3D and assembles them into rough initial masks. Secondly, these initial masks are refined using RGB. Surprisingly, our framework is able to learn from synthetic RGB-D data where the RGB is non-photorealistic. To train our method, we introduce a large-scale synthetic dataset of random objects on tabletops. We show that our method can produce sharp and accurate segmentation masks, outperforming state-of-the-art methods on unseen object instance segmentation. We also show that our method can segment unseen objects for robot grasping.

I. INTRODUCTION

The paper addresses unseen object instance segmentation for tabletop robots despite limited real-world data, proposing UOIS-Net and a synthetic training dataset that separate depth-based initialization from RGB refinement.

  • Motivation: Robots need to recognize and segment previously unseen objects, but large-scale annotated datasets covering diverse objects are expensive and generally unavailable.This motivates synthetic training data for robotic perception.
  • Motivation: Synthetic RGB creates a sim-to-real gap because non-photorealistic images transfer poorly, while photorealistic rendering is computationally expensive.Synthetic depth has shown better generalization in simpler settings, motivating separate modality use.
  • Method: UOIS-Net first uses depth to predict 2D or 3D object-center votes and assemble rough masks, then uses RGB to refine their boundaries.The two stages are a Depth Seeding Network and a Region Refinement Network.
  • Results: Conditioned on initial masks, the RGB refinement network trains on non-photorealistic synthetic images and generalizes robustly to cluttered real-world objects.The refinement network works almost as well as when trained on real data and produces sharp masks despite noisy depth.
  • Dataset: The authors introduce the Tabletop Object Dataset, a large synthetic collection of random ShapeNet objects and tables simulated with PyBullet.The dataset provides synthetic depth and non-photorealistic RGB for training.
  • Contributions: UOIS-Net extends prior work with a 3D depth-seeding architecture and a separation loss that improves center-vote clustering in cluttered scenes.The paper reports that the separation loss is crucial for strong performance.

B. Instance-level Object Segmentation

The paper targets class-agnostic instance masks for arbitrary tabletop objects, combining depth-based mask generation with RGB refinement to support unseen-object interaction.

  • Related Work: Top-down methods can struggle when bounding boxes contain multiple objects, motivating bottom-up or class-agnostic alternatives for cluttered scenes.Class-agnostic proposal methods may segment everything and require post-processing to select relevant masks.
  • Task Definition: Unlike category-level instance segmentation, the task produces masks without semantic labels for arbitrary objects, including unseen categories.The masks are intended for downstream robotic grasping and manipulation.
  • Method: UOIS-Net processes depth and RGB separately: DSN generates initial masks, IMP robustifies them, and RRN snaps noisy boundaries to RGB object edges.The DSN and RRN are trained separately because initial-mask construction includes non-differentiable operations.
  • Sim-to-Real: Both networks train fully in simulation without real-data fine-tuning, yet the framework generalizes to real scenes and enables robotic tasks involving unseen objects.The reported motivation is to exploit abundant simulated scenes without manual annotation.

IV. DEPTH SEEDING NETWORK

The Depth Seeding Network uses organized 3D depth information to predict foreground structure and center directions, then converts votes into initial instance masks through Hough voting.

  • Network Architecture: The DSN takes an organized XYZ point cloud derived from depth and produces class-agnostic initial instance segmentation masks.Depth is used first because it generalizes reasonably well in sim-to-real settings.
  • 2D and 3D DSN: The paper compares 2D and 3D center prediction because 2D voting has limitations that motivate a novel architecture reasoning directly in 3D space.The 3D formulation is presented as an extension intended to address those shortcomings.
  • Network Architecture: The 2D DSN predicts a semantic mask with background, tabletop, and object classes plus a unit vector from each pixel toward its observable object center.The network uses a U-Net encoder-decoder with separate foreground and center-direction branches.
  • Mask Construction: Hough voting aggregates foreground-pixel directions, thresholds candidate centers, applies non-maximum suppression, and assigns pixels to selected centers.The resulting assignments form the initial instance masks.
  • Mask Construction: Hough voting uses inlier, distance, and percentage thresholds to reject candidate centers lacking sufficient directional support and reduce false positives.Non-maximum suppression selects local maxima among the vote scores.

2) Loss Functions:

The depth-seeding network combines foreground segmentation, direction prediction, and 2D/3D center voting to form initial object masks. Its 3D formulation addresses occluded centers and uses clustering-oriented design choices for cluttered scenes.

  • 2) Loss Functions:: The DSN uses weighted cross-entropy for semantic foreground prediction and weighted cosine similarity for direction prediction.Direction loss also constrains background and tabletop pixels to point in a fixed direction to reduce false positives.
  • 2) Loss Functions:: 3D reasoning mitigates 2D failures when an object center is occluded by another object.The 3D DSN predicts offsets in XYZ space rather than 2D directions.
  • 2) Loss Functions:: The depth-seeding network predicts foreground pixels and center votes from organized depth or point-cloud inputs, then clusters votes to obtain initial masks.The 2D version predicts directions, while the 3D version predicts offsets whose sums with point coordinates are center votes.
  • 2) Loss Functions:: Dilated ESP modules increase the DSN receptive field while retaining a lightweight spatial-pyramid design.The sixth, eighth, and tenth convolution layers are replaced with ESP modules.
  • 2) Loss Functions:: Mean-shift clustering assigns foreground pixels to modes of their 3D center-vote distribution without requiring the object count beforehand.The Gaussian-kernel bandwidth σ controls the number of modes and is especially important when objects are close together.

2) Loss Functions:

The 3D DSN is trained with foreground, center-offset, clustering, and separation losses. These objectives make center votes clusterable while separating votes from neighboring objects in clutter.

  • 2) Loss Functions:: The 3D DSN combines foreground, center-offset, clustering, and separation losses in its total training objective.The total loss is λ_fgℓ_fg + λ_coℓ_co + λ_clℓ_cl + λ_sepℓ_sep.
  • 2) Loss Functions:: Center-offset learning uses a Huber loss to move predicted votes toward ground-truth 3D object centers with inverse instance-size weighting.Centers outside the camera view are projected into the camera’s field of view.
  • 2) Loss Functions:: The clustering loss unrolls several Gaussian mean-shift iterations and encourages points to remain close to their cluster while separating from other clusters.The mean-shift iteration is treated as a parameter-free network layer, and stochastic sampling reduces memory usage.
  • 2) Loss Functions:: The separation loss encourages center votes to remain near their own object center while staying far from other object centers.This makes post-processing Gaussian mean-shift clustering easier, particularly when object centers are close in heavy clutter.

V. INITIAL MASK PROCESSING MODULE

The initial mask processing module cleans noisy DSN masks before RGB refinement. It removes isolated noise, closes small holes, and retains only the largest connected component per instance.

  • V. INITIAL MASK PROCESSING MODULE: Initial DSN masks can contain salt-and-pepper noise and holes that hinder the region refinement network.These errors arise in masks computed from foreground and center-vote predictions.
  • V. INITIAL MASK PROCESSING MODULE: Opening, closing, and largest-component selection robustify each instance mask before refinement.Opening removes isolated noise, closing fills small holes, and smaller disconnected components are discarded.

VI. REGION REFINEMENT NETWORK

The region refinement network uses RGB to correct depth-derived mask boundaries, while training relies on a synthetic tabletop dataset and perturbed masks. Despite non-photorealistic RGB, the refined masks can follow object boundaries.

  • VI. REGION REFINEMENT NETWORK: The RRN is designed to snap noisy depth-based mask edges to object boundaries using RGB.Depth generalizes reasonably well from simulation to reality, but sensor noise leaves boundary errors for RGB refinement to correct.
  • VI. REGION REFINEMENT NETWORK: The region refinement network combines a cropped RGB patch with an initial mask to output a refined instance mask.Its four-channel input is resized to 224 × 224, and the output mask probabilities are thresholded.
  • VI. REGION REFINEMENT NETWORK: RRN training uses perturbed ground-truth masks because synthetic DSN outputs are typically too clean for effective training.The perturbations emulate realistic initial-mask errors through translation, rotation, adding, cutting, morphology, and random ellipses.
  • VI. REGION REFINEMENT NETWORK: The Tabletop Object Dataset contains 40k synthetic scenes of cluttered ShapeNet objects placed on tables in SUNCG home environments.Scenes vary object placement, stacking, physics-based settling, and camera views.
  • VI. REGION REFINEMENT NETWORK: Despite non-photorealistic RGB caused partly by limited PyBullet texture rendering, the RRN learns to align masks with object boundaries.The dataset includes heavily cluttered scenes and RGB images with visibly unrealistic rendering.

VIII. EXPERIMENTS

The experiments evaluate UOIS-Net variants and baselines under specified optimization, augmentation, and inference settings. The pipeline combines separately configured 2D and 3D depth networks with an RGB refinement network.

  • UOIS-Net-2D and UOIS-Net-3D are evaluated against Mask R-CNN and PointGroup on real datasets.The variant names indicate whether the depth voting network reasons in 2D or 3D.
  • 2D depth networks use SGD for 100k iterations, learning rate 1e-2, batch size 8, and discretized Hough voting.The Hough voting layer uses 100 angular bins and processes every tenth pixel for efficiency.
  • 3D depth networks use Adam for 150k iterations with learning rate 1e-4, batch size 8, and separate clustering settings for training and testing.Training uses 5 cluster-loss rollouts and 150 seeds, while testing uses 10 rollouts and 200 seeds.
  • Depth training augments inputs with multiplicative gamma noise and Gaussian Process noise on backprojected point clouds.
  • Region Refinement Networks use SGD for 100k iterations with learning rate 1e-2 and batch size 16, while the full pipeline runs at approximately 3-5 frames per second.RRN inputs are padded by 25% of the initial mask bounding-box size in each dimension.
  • Baseline training follows the official Detectron schedule for Mask R-CNN and uses 300k iterations with batch size 4 for PointGroup.Semantic-score clustering is removed because the task has only one meaningful semantic class, foreground.

B. Datasets

The evaluation uses two real-world datasets and an additional in-the-wild RGB dataset, with overlap and boundary precision/recall/F-measures designed to assess segmentation quality. Boundary metrics specifically account for annotation noise and boundary sharpness.

  • Datasets: The real-world evaluation uses OCID with 2,346 images and OSD with 111 manually labeled images.OSD annotations are manually produced, whereas OCID labels are semi-automatically constructed and can have noisy boundaries.
  • Datasets: The OID experiment uses roughly 220k real RGB instance masks from 156 tabletop-relevant classes to test the RRN’s Sim-to-Real gap.The filtered data comes from approximately 9 million in-the-wild images and 2.8 million segmentations across 350 classes.
  • Metrics: Overlap P/R/F metrics match predicted and ground-truth masks with the Hungarian method and penalize false positives.The measures report precision, recall, and F-measure for matched object masks.
  • Metrics: Boundary P/R/F complements overlap scores because fuzzy and sharp boundaries can receive similar overlap measurements.It applies dilation-based slack to reduce sensitivity to noisy manual or semi-automatic boundary annotations.
  • Metrics: Boundary evaluation uses the same Hungarian matching while treating predicted and ground-truth object boundaries as the matched sets.The dilation operation uses a circular kernel whose diameter depends on image size.
  • Metrics: All reported precision, recall, and F-measure values are scaled to the range [0, 100].

D. 2D Quantitative Results

UOIS-Net-2D outperforms depth- and RGB-based baselines despite training without real data, while its separate use of depth and RGB supports sharper masks and stronger real-world generalization.

  • UOIS-Net-2D significantly outperforms real-data-trained baselines on F-measure despite never seeing real data.V4R has a substantial advantage because it was trained on OSD, whose distribution closely matches OCID.
  • UOIS-Net-2D outperforms Mask R-CNN on OSD and slightly on OCID, while achieving comparable performance to PointGroup.Both Mask R-CNN and PointGroup are trained on RGB-D from TOD.
  • Removing RRN improves UOIS-Net performance on noisy-boundary OCID labels, but reduces UOIS-Net-2D performance by almost 20% relatively on manually annotated OSD.The authors attribute this contrast to OCID’s noisy label boundaries; refined qualitative masks can therefore have worse quantitative scores there.
  • Separately leveraging depth and RGB yields better OSD results than directly training Mask R-CNN or DSN on synthetic RGB-D inputs.Synthetic RGB alone generalizes poorly, while depth substantially improves generalization; directly training DSN on RGB-D causes a performance drop.
  • RRN refinement has similar overlap performance when trained on TOD or real OID images, with only slightly better boundary measures from real-image training.The OID-trained RRN provides an approximate upper bound for synthetically trained refinement.
  • The IMP is crucial because RRN refinement hurts raw DSN masks but substantially improves boundary precision, recall, and F-measure after morphological cleanup.The cleanup applies opening, closing, and closest-connected-component selection to remove noise and holes before refinement.

F. 3D Quantitative Results

UOIS-Net-3D improves unseen-object segmentation over UOIS-Net-2D and strong baselines, with gains attributed primarily to 3D center voting and better initial masks. Ablations show that separation loss and receptive-field expansion are important, while clustering parameters require tuning for scene clutter.

  • Comparison to baselines: UOIS-Net-3D increases recall over UOIS-Net-2D, producing relative gains of 4.5% in overlap F-measure and 5.5% in boundary F-measure.The comparison attributes the additional recall mainly to 3D center voting and better initial masks.
  • Comparison to SOTA: UOIS-Net-3D raises overlap F-measure by 5.8% and boundary F-measure by 6.7% over UOIS-Net-2D on OCID.It also exceeds Mask R-CNN by 8.1% and 6.0%, and PointGroup by 7.9% and 6.3%, on the respective metrics.
  • Loss function ablation: Adding ℓsep improves overlap F-measure by 10.1% and boundary F-measure by 14.4% over the base loss model.The separation loss pushes center votes from different objects apart, making Gaussian mean shift clustering easier in clutter.
  • ESP module ablation: Using an ESP module improves UOIS-Net-3D by 3.0% on overlap F-measure and 3.5% on boundary F-measure on OCID.The ablation evaluates the module as a way to provide a higher receptive field.
  • Hyperparameter ablations: The best tested σ is 0.02 for both OCID and OSD, while τ-based metrics plateau after τ reaches 15.Small σ values oversegment objects, whereas large values can merge nearby objects; the appropriate setting depends on clutter.

G. 3D Qualitative Results

Qualitative comparisons show that 3D center reasoning corrects several 2D voting failures, especially for occluded, thin, and small objects. Separation loss produces tighter, better-separated votes, but depth ambiguity, nonconvexity, and occlusion remain failure modes.

  • 2D vs. 3D comparison: 3D center voting corrects occluded-center failures and detects more small, thin objects than 2D voting.The qualitative examples include pens and bananas, for which discretized 2D directions can fail.
  • Center votes: ℓsep makes center votes tighter and more separated, enabling correct clustering of nearby fruits and objects with close 3D centers.Without ℓsep, votes are spread or jumbled, making post-processing clustering difficult.
  • Failure modes: Objects close together can be under-segmented when their depth-derived center votes are too close for post-processing clustering.Flat, aligned cereal boxes are especially difficult from depth alone, making appearance important for separation.
  • Failure modes: Highly nonconvex objects can be over-segmented, and masks split by occlusion can remain problematic even with 3D reasoning.The examples include power drills and objects whose visible regions are divided by an occluding object.

H. Quantifying Generalization from Sim to Real

The paper evaluates simulation-to-real generalization on a synthetic held-out test set and demonstrates a robotic clearing task using segmentation-driven grasping. Real-world comparisons favor UOIS-Net, although competing methods outperform it on the synthetic test set.

  • Simulation-to-real evaluation: Mask R-CNN and PointGroup outperform UOIS-Net on all metrics of the 20k-image TOD test set.The test uses object instances absent from training, providing a held-out synthetic generalization comparison.
  • Simulation-to-real evaluation: On real-world data, UOIS-Net outperforms Mask R-CNN and PointGroup despite the reverse ordering on the synthetic TOD test set.This contrast is used to assess generalization from simulation to real-world scenes.
  • Robotic manipulation: The system segments cluttered tabletop objects and supplies the closest object’s point cloud to 6-DOF GraspNet for grasp generation.Other objects are treated as obstacles during the robot’s clearing task.
  • Conclusion: The conclusion reports that UOIS-Net produces strong unseen-object segmentation and supports robotic tasks despite training only on non-photorealistic simulated data.The method is described as using separate RGB and depth processing to produce sharp masks.
Loading 2007.08073v2…