Source-linked AI summary
CNOS: A Strong Baseline for CAD-based Novel Object Segmentation
Van Nguyen Nguyen, Thibault Groueix, Georgy Ponimatkin, Vincent Lepetit, Tomas Hodan
TL;DR
Supervised pose estimation requires extensive target-specific training, making it burdensome to introduce unseen objects. CNOS renders CAD templates and matches their DINOv2 descriptors to SAM/FastSAM proposals from RGB images. It achieves state-of-the-art results on the seven core BOP datasets, including an absolute 19.8% AP improvement over Chen et al.
Problem
Supervised pose estimation relies on extensive target-specific training data, making introducing new objects costly in data synthesis or annotation and retraining.
Source: §1 · p. 1Method
CNOS renders CAD templates described by DINOv2 and matches their descriptors to SAM or FastSAM proposals generated from an RGB image.
Source: §1 · p. 1Results
CNOS achieves state-of-the-art performance on the seven core BOP datasets, surpassing Chen et al. by absolute 19.8% AP and Mask R-CNN.
Source: §1 · p. 2, §4.2 · p. 5Takeaways & Limitations
The authors position training-free CNOS as a baseline for CAD-based novel object segmentation and an initial stage for novel-object pose-estimation pipelines.
Source: §5 · p. 6Most segmentation mistakes arise from erroneous DINOv2-based proposal classification, which the authors identify as needing improvement to close the gap with ZebraPose.
Source: §4.2 · p. 5
Abstract
from arXiv · showhide
We propose a simple three-stage approach to segment unseen objects in RGB images using their CAD models. Leveraging recent powerful foundation models, DINOv2 and Segment Anything, we create descriptors and generate proposals, including binary masks for a given input RGB image. By matching proposals with reference descriptors created from CAD models, we achieve precise object ID assignment along with modal masks. We experimentally demonstrate that our method achieves state-of-the-art results in CAD-based novel object segmentation, surpassing existing approaches on the seven core datasets of the BOP challenge by 19.8% AP using the same BOP evaluation protocol. Our source code is available at https://github.com/nv-nguyen/cnos.
1. Introduction
Supervised pose-estimation methods require extensive target-specific training, making it burdensome to introduce new products. CNOS instead segments unseen objects from CAD models and outperforms unsupervised and supervised comparators on the seven core BOP datasets.
1. Introduction
- Supervised pose-estimation methods rely on extensive target-specific training data, making it burdensome to introduce new objects such as warehouse products.Introducing an unseen object can require synthesizing or annotating data and retraining the model.Source: §1 · p. 1
- CNOS outperforms both the unsupervised method of Chen et al. and supervised Mask R-CNN on the seven core BOP datasets.Mask R-CNN was trained on tens of thousands of images per dataset; CNOS runtime, like Mask R-CNN's, is dominated by proposal generation.Source: Figure 1 · p. 1, §1 · p. 2
- CNOS segments unseen objects using only their CAD models, rendering templates described by DINOv2 and matching them to SAM or FastSAM proposals from an RGB image.Rendering templates takes less than two seconds per CAD model, compared with several hours typically needed to retrain supervised methods.Source: §1 · p. 1
2. Related work
Prior segmentation methods often depend on target-specific training, depth or stereo input, or initial bounding boxes. This work focuses on segmenting unseen objects from CAD models in a single RGB image.
Related work
- Seen-object segmenters such as Mask R-CNN are fine-tuned on target-specific data and cannot handle new objects without retraining.Such supervised methods have been shown robust under heavy occlusion and lighting variation.Source: §2 · p. 2
- Prior unseen-class work includes class-agnostic networks that generalize to new classes [6] [7], saliency-based novel-class discovery [29], and tracking that assumes an initial bounding box [18].The tracking approach propagates the box across frames using optical flow before predicting masks.Source: §2 · p. 2
- UOIS-Net [27] and Xiang et al. [26] use RGB-D input, while Durner et al. [8] use stereo RGB; CNOS instead targets a single RGB image and CAD models.UOIS-Net assembles masks from depth-based center votes and refines them with RGB; Xiang et al. use learned embeddings and mean-shift clustering.Source: §2 · p. 2
- CNOS surpasses the SAM/ImageBlind CAD-segmentation method of Chen et al. by an absolute 19.8% AP.Chen et al. extract proposals with SAM and combine them with visual clues from ImageBlind [9].Source: §2 · p. 2
3. Method
CNOS renders CAD templates and describes them with DINOv2, generates masks from an RGB image, then matches proposal descriptors to templates to assign object IDs and confidence scores.
3. Method
- CNOS has three stages: onboard CAD models with rendered descriptors, generate image proposals and descriptors, then match proposals to CAD templates to retrieve and label object masks.The matching stage uses CAD-model visual descriptors to assign labels to proposals.Source: §3 · p. 2
3.1. Onboarding stage
- The onboarding stage renders RGB templates from 42 CAD-model viewpoints by default and uses DINOv2 cls tokens as visual descriptors.The viewpoints are defined by an icosphere; the default descriptor dimension is 1024.Source: §3.1 · p. 2
3.2. Proposal stage
- The proposal stage uses SAM or FastSAM to generate a variable number of unlabeled masks, then removes background, crops, and standardizes proposals for batched DINOv2 description.Each standardized proposal is resized to 224×224 before its DINOv2 cls token is extracted.Source: §3.2 · pp. 2–3
3.3. Matching stage
- The matching stage compares proposal and template descriptors by cosine similarity, aggregates scores across views using the best-performing Mean of top k, and assigns IDs and confidence scores by max and argmax.The resulting labeled proposals contain modal masks covering visible object surfaces; a confidence threshold can filter incorrect labels.Source: §3.3 · p. 3
4. Experiments
Across the seven core BOP datasets, CNOS outperforms the prior CAD-based method and surpasses target-trained Mask R-CNN. The experiments examine rendering, viewpoints, similarity aggregation, runtime, and possible extensions.
4.2. Comparison with the state of the art
- CNOS beats Chen et al. by 19.8% absolute AP and surpasses target-trained Mask R-CNN across the seven BOP core datasets.The comparison uses the BOP evaluation protocol; Table 1 distinguishes supervised and unsupervised methods.Source: Table 1 · p. 4, §4.2 · p. 5
4.1. Experimental setup
- Evaluation covers 132 objects in cluttered, occluded scenes across seven BOP datasets, using COCO-style AP averaged over IoU thresholds from 0.50 to 0.95.Source: §4.1 · pp. 3–4
4.3. Ablation study
- Realistic rendering improves AP by 4.3% compared with Pyrender rendering.The paper attributes the gain to reducing the domain gap between synthetic and real images.Source: §4.3 · p. 5
- Denser viewpoint sampling does not improve performance over the coarse 42-view set.The authors attribute this to sufficient coverage from the existing viewpoints.Source: §4.3 · p. 5, Table 3 · p. 6
- Among tested similarity aggregators, Meank (k=5), averaging the five highest similarities, performs best.Source: §4.3 · p. 5, Table 4 · p. 6
- Onboarding is bottlenecked by template generation, while proposal-stage runtime is bottlenecked by the segmentation algorithm.Onboarding is performed once per CAD model.Source: §4.3 · p. 6
4.4. Discussion
- DINOv2 cls tokens appear useful for object identity but not pose initialization in the proposed pipeline.The attempted pose-initialization approach did not yield successful results.Source: §4.4 · p. 6
- CNOS could be adapted to CAD-free one-shot or few-shot segmentation by extracting reference descriptors from available images and leaving the rest of the pipeline unchanged.Source: §4.4 · p. 6
5. Conclusion
CNOS segments novel objects from CAD models without training and achieves accuracy comparable to prior supervised methods trained on large annotated datasets. The authors propose it as a baseline for CAD-based novel object segmentation and as an initial stage for novel-object pose-estimation pipelines.
5. Conclusion
- CNOS achieves accuracy comparable to prior supervised methods trained on large-scale annotated datasets.The method uses CAD models without training.Source: §5 · p. 6
- The method segments novel objects solely from their CAD models, without training.Source: §5 · p. 6
- The authors hope CNOS will serve as a standard baseline for CAD-based novel object segmentation and an initial stage in novel-object pose-estimation pipelines.Source: §5 · p. 6