PaperlayerView exact revision on arXiv

Source-linked AI summary

Scan2CAD: Learning CAD Model Alignment in RGB-D Scans

Armen Avetisyan, Manuel Dahnert, Angela Dai, Manolis Savva, Angel X. Chang, Matthias Nießner

arXiv:1811.11187v1cs.CV

TL;DR

Scan2CAD addresses the challenge of aligning clean CAD models with noisy, incomplete RGB-D scans. It introduces a large alignment dataset and combines learned correspondence heatmaps with variational 9DoF optimization, outperforming handcrafted and CNN-based baselines by 21.39%.

  • Problem

    Aligning clean CAD models to noisy, incomplete scans is difficult because their low-level geometric features differ, while existing datasets lack fine-grained alignments at scale.

  • Method

    Scan2CAD constructs scan-to-CAD alignment data and uses a 3D CNN for correspondence heatmaps followed by variational optimization of 9DoF poses.

  • Results

    21.39% improvement over handcrafted feature descriptors and state-of-the-art CNN-based methods is reported on the Scan2CAD benchmark.

  • Takeaways & Limitations

    The method transforms noisy, incomplete RGB-D scans into a clean, complete CAD model representation through scan-to-CAD alignment.

  • Takeaways & Limitations

    CAD model retrieval uses a basic component requiring exhaustive search, and the method currently relies only on geometric information.

Abstract

from arXiv · show

We present Scan2CAD, a novel data-driven method that learns to align clean 3D CAD models from a shape database to the noisy and incomplete geometry of a commodity RGB-D scan. For a 3D reconstruction of an indoor scene, our method takes as input a set of CAD models, and predicts a 9DoF pose that aligns each model to the underlying scan geometry. To tackle this problem, we create a new scan-to-CAD alignment dataset based on 1506 ScanNet scans with 97607 annotated keypoint pairs between 14225 CAD models from ShapeNet and their counterpart objects in the scans. Our method selects a set of representative keypoints in a 3D scan for which we find correspondences to the CAD geometry. To this end, we design a novel 3D CNN architecture that learns a joint embedding between real and synthetic objects, and from this predicts a correspondence heatmap. Based on these correspondence heatmaps, we formulate a variational energy minimization that aligns a given set of CAD models to the reconstruction. We evaluate our approach on our newly introduced Scan2CAD benchmark where we outperform both handcrafted feature descriptor as well as state-of-the-art CNN based methods by 21.39%.

1. Introduction

Indoor RGB-D scans remain noisy and incomplete despite advances in reconstruction, motivating scan-to-CAD alignment. Scan2CAD addresses this gap with a large dataset, a joint real-synthetic 3D CNN, and variational alignment optimization.

  • Motivation: RGB-D reconstructions remain noisy and incomplete because of sensor noise, motion blur, and scanning patterns.Existing completion methods also struggle to recover sharp edges and planar surfaces comparable to artist-modeled 3D content.
  • Motivation: Aligning CAD models to scans is difficult because similar high-level structures can have substantially different low-level geometric features.This limits handcrafted descriptors and highlights the need for appropriate training data for deep learning approaches.
  • Contributions: A novel 3D CNN learns a joint embedding between real and synthetic objects to predict correspondence heatmaps.The heatmaps identify likely correspondences between scan geometry and CAD geometry.
  • Contributions: A variational optimization formulation uses correspondence heatmaps to minimize distances between scan keypoints and CAD correspondences, producing robust 9DoF alignments.This optimization follows the learned correspondence prediction stage.

2. Related work

Prior work covers RGB-D reconstruction, geometric descriptors, learned matching, and CAD retrieval, but existing evaluations and datasets do not adequately measure fine-grained scan-to-CAD alignment at scale.

  • RGB-D Scanning and Reconstruction: RGB-D reconstruction research has advanced through volumetric fusion, octree or hash-based representations, point fusion, and global pose alignment.These methods establish the reconstruction setting in which scan-to-CAD alignment is needed.
  • 3D Features for Shape Alignment and Retrieval: Handcrafted geometric descriptors and learned feature methods have been used for shape retrieval, alignment, segmentation, and scan keypoint matching.Examples include Spin Images, FPFH, PPF, random forests, Siamese networks, and view-consistency losses.
  • Related CAD Retrieval: Methods for retrieving and aligning CAD models from RGB or RGB-D images differ from Scan2CAD’s focus on geometric alignment independent of RGB information.The distinction is between CAD-to-image methods and scan-to-CAD geometric alignment.
  • Shape Retrieval Challenges and RGB-D Datasets: Existing 3DOR evaluations treat relevance mainly as binary category or sub-category matching, without measuring alignment quality.This leaves pose accuracy between scan objects and CAD models unaddressed.
  • Shape Retrieval Challenges and RGB-D Datasets: ScanNet provides aligned CAD models for only 200 of 36000 annotated objects, with reported category mismatches and alignment errors.Other datasets target coarse poses or CAD-to-2D-image alignment rather than the same fine-grained task.
  • Shape Retrieval Challenges and RGB-D Datasets: No existing dataset provides fine-grained object-instance alignments at Scan2CAD’s scale across 14225 CAD models and 1506 scans.This motivates the Scan2CAD dataset and benchmark.

3. Overview

Scan2CAD aligns candidate CAD models to noisy scans by predicting scan-to-CAD correspondences and optimizing 9DoF transformations. Its overview combines dataset annotation, learned heatmap prediction, and variational alignment with confidence-based pruning.

  • Task: The task is to estimate translation, rotation, and scale for each CAD model so it aligns with a semantically matching scan object.The transformation has 9DoF, with three degrees each for translation, rotation, and scale.
  • Dataset and Benchmark: The dataset annotation pipeline selects a geometrically similar CAD model, defines scan-to-model keypoint pairs, and computes ground-truth 9DoF alignments.The pipeline is applied across ScanNet’s original training, validation, and test splits.
  • Heatmap Prediction Network: A 3D CNN processes a scan volume around a candidate keypoint and a volumetric CAD representation to predict a correspondence heatmap over CAD voxels.Classification produces sparse correspondences for subsequent pose optimization.
  • Dataset and Benchmark: The annotation interface first retrieves class-matching CAD models after a scan anchor is placed, then records paired scan and CAD keypoints for ground-truth transformation estimation.The two-step interface supports object retrieval followed by correspondence annotation.
  • Alignment Optimization: Harris scan keypoints seed correspondence prediction, variational optimization finds 9DoF transformations, and a geometric confidence metric prunes false alignments.The pipeline uses predicted heatmaps for each Harris keypoint and CAD model.

4. Dataset

Scan2CAD builds a verified scan-to-CAD dataset through structured annotation of ShapeNet models against ScanNet objects. It records keypoints, transformations, symmetries, and benchmark criteria for evaluating alignment quality.

  • Data Representation: Each ScanNet scene object is matched with a ShapeNet CAD model, shared keypoint correspondences, and one alignment transformation matrix.ShapeNet’s canonical orientation also induces an amodal tight oriented bounding box for each scan object.
  • Data Annotation: The web annotation application separates object retrieval from alignment, allowing annotation jobs to be scaled and distributed.Retrieval begins by selecting a scan surface point and displaying CAD models matching the inferred category.
  • Data Annotation: At least 6 paired keypoints are required before alignment computation, which uses CMA-ES to minimize point-to-point distance.Annotators alternately select corresponding points on the CAD model and scan object.
  • Dataset Statistics: 97607 keypoint pairs cover 14225 CAD models, 3049 unique models, and 1506 scans; approximately 28% of unique models receive symmetry tags.Symmetry labels include C2, C4, and C∞ rotational symmetries.
  • Dataset Statistics: Seven part-time annotators and author-led verification passes supported dataset quality, with each scene edited 1.76 times on average.Each object took 2.48 minutes to align on average, while complete scenes averaged 20.52 minutes.
  • Dataset Statistics: Annotated scan keypoints averaged 3.5cm from their CAD counterparts, compared with 12.8cm for the nearest Harris keypoints.The comparison distinguishes manually placed correspondences from detector-based keypoints.
  • Benchmark: A benchmark counts alignment as successful only when category matches and translation, rotation, and scale errors remain within specified bounds.Exact CAD-instance matching is not required because ShapeNet models may not identically match scanned objects.
  • Benchmark: The dataset includes symmetry annotations and an automated test script operating on a hidden test set for standardized future comparisons.This supports consistent benchmark evaluation by subsequent methods.

5. Correspondence Prediction Network

The correspondence network encodes scan SDFs and CAD distance fields into a shared embedding, predicting heatmaps, semantic compatibility, and scale for scan-to-CAD alignment.

  • Network Architecture: The decoder predicts a CAD-space correspondence heatmap, semantic compatibility probability, and CAD-to-scan scale.These outputs support correspondence identification and later alignment optimization.
  • Data Representation: The network takes scan signed distance fields and CAD distance fields as paired voxel-grid inputs.Scan volumes use SDFs from RGB-D reconstruction, while CAD models use unsigned distance fields.
  • Network Architecture: Separate 3D CNN encoders compress the scan and CAD volumes before concatenation and decoding.The encoded feature volumes are concatenated before the decoder predicts the three outputs.
  • Heatmap Prediction: The final heatmap multiplies sigmoid- and softmax-based predictions to balance recall with locally sharp keypoint localization.The loss emphasizes sparse positive keypoint voxels, which comprise approximately 1% of target voxels.
  • Training: The model is trained with a weighted combination of compatibility, scale, and heatmap losses, masking heatmap and scale losses to positive samples.The CAD encoder is separately pre-trained as an autoencoder on ShapeNet models.
  • Training: Training uses annotated, augmented, and negative scan-CAD pairs, producing a 1:2 positive-to-negative ratio.Augmented positives are generated by sampling CAD surface points and rejecting projections at least 3 cm from the scan surface.

6. Alignment Optimization

The alignment stage filters scan-CAD correspondences, minimizes a heatmap-based variational objective over pose and scale, and uses multi-start refinement and confidence pruning.

  • Filtering: Alignment optimization begins with Harris keypoints whose predicted scan-CAD correspondences form the optimization input.Each scan keypoint is tested against every candidate CAD model.
  • Filtering: Compatibility scores reject semantically mismatched correspondences using Otsu thresholding, typically retaining about 10% of the original pairs.The adaptive threshold was more effective in practice than a fixed threshold.
  • Variational Optimization: The method constructs point-heatmap pairs and minimizes their alignment energy over Lie-algebra pose coordinates and scale.The transformation includes translation, rotation, and scale, with scale regularization to prevent very large degenerate solutions.
  • Variational Optimization: Levenberg-Marquardt solves the objective through a coarse-to-fine heatmap scale pyramid to mitigate zero gradients from poor initialization.The pyramid is especially useful when initial poses are far from the solution.
  • Multi-start Alignment: Restarting optimization from different translations yields multiple alignments per CAD model, accommodating repeated objects in a scene.The procedure generates as many alignments as required for evaluation.
  • Pruning: A confidence metric prunes potential misalignments after optimization.The supplied material identifies pruning as the final selection stage but refers detailed confidence construction to the appendix.

7. Results

Scan2CAD’s learned cross-domain features and alignment optimization improve correspondence prediction and scan-to-CAD alignment, including an unconstrained setting with randomly selected CAD candidates.

  • 7.1. Correspondence Prediction: 63.94%: the best 3D CNN achieves this voxel-wise F1-score for correspondence heatmap prediction.The evaluation uses Gaussian-blurred targets in a highly sparse voxel setting, where approximately 99% of target voxels are zero-valued.
  • 7.1. Correspondence Prediction: Compatibility strongly improves heatmap prediction by efficiently detecting false correspondences, while symmetry awareness also yields significant improvement.Predicting scale has little effect on heatmaps but becomes effective during the later alignment stage.
  • 7.2. Alignment: 31.68%: the best method achieves this accuracy on the CAD alignment benchmark, significantly outperforming handcrafted descriptors and 3DMatch.The comparison combines feature descriptors with RANSAC-based pose estimation; Scan2CAD’s explicit real-synthetic learning and alignment optimization produce more reliable alignments.
  • 7.2. Alignment: Qualitative results across four test ScanNet scenes show more reliable keypoint correspondences and significantly more accurate alignments.The reported improvement links learned geometric features with the subsequent alignment optimization.
  • 7.2. Alignment: 400 randomly selected ShapeNetCore CAD models define an unconstrained scenario that more closely mimics real-world use.This setting removes the assumption that a ground-truth CAD model set is provided.

8. Limitations

The method provides only a basic retrieval component, requiring exhaustive search over candidate CAD models; it also currently uses geometry without learned color features.

  • 8. Limitations: CAD model retrieval remains limited to a basic component requiring exhaustive search over the candidate set.The authors identify efficient shape-similarity retrieval as an immediate next step.
  • 8. Limitations: The current method uses only geometric information, leaving learned color features as a possible way to improve alignment results.The authors note that RGB data is typically higher-resolution than depth or geometry.

9. Conclusion

Scan2CAD combines a new scan-to-CAD alignment dataset, correspondence heatmap prediction, and variational energy minimization to align CAD models with noisy RGB-D scans. The full training corpus improves generalization, while alignment sensitivity varies across translation, rotation, and scale thresholds.

  • 9. Conclusion: The method predicts correspondence heatmaps with a 3D CNN and optimizes those predictions through variational energy minimization to find 9DoF CAD-to-scan poses.This pipeline targets transformation of noisy, incomplete RGB-D scans into clean, complete CAD model representations.
  • Training curves: Full-dataset training mitigates overfitting, whereas half- and quarter-dataset experiments show severe overfitting and poorer generalization.The comparison uses training and validation curves for full, half, and quarter dataset sizes.
  • Correspondence predictions: Correspondence predictions include symmetry-equivalent matches, while a 3cm voxel grid and 64-cell dimension provide a 1.92m receptive extent per dimension.Larger objects require a larger receptive field to infer correspondences from more global semantic context.
  • Pruning: The alignment pipeline prunes objects using scan free space, then removes lower-confidence models within 0.3m of higher-ranked alignments.The free-space pruning step is particularly important when the candidate CAD set is not provided in the unconstrained setting.

E. Alignment Error Analysis

The alignment results respond differently to translation, rotation, and scale thresholds. Threshold sensitivity is assessed by varying one parameter block while holding the other two at default values.

  • Threshold sensitivity: Translation, rotation, and scale have different threshold sensitivities in the alignment results.The analysis gauges each parameter block independently by varying one threshold at a time.
  • Default thresholds: The default thresholds are ϵt = 0.2m, ϵr = 20°, and ϵs = 20%.The remaining thresholds are held fixed at these defaults when one parameter block is varied.

F. Baseline Method Details

Baseline methods compare handcrafted and learned geometric descriptors for scan-to-CAD correspondence, followed by RANSAC-based alignment. Their parameters are selected through a validation-set hyperparameter search.

  • Handcrafted descriptors: FPFH and SHOT compute geometric point properties within support regions around keypoints using point-cloud descriptors.Both are implemented through the Point Cloud Library.
  • Handcrafted descriptors: Li et al. incorporate free space around keypoints into descriptor distances between scan and CAD points.The authors adapt the original implementation to the shared evaluation framework and omit geometric-primitive computation because real-time constraints are unnecessary.
  • Learned descriptors: 3DMatch converts a truncated-distance-function volumetric patch around each keypoint into a 512-dimensional feature vector using 3D convolutions and max-pooling.The network is trained with a contrastive loss and applied separately to detected scan and CAD keypoints.
  • Correspondence generation: Candidate correspondences require a height difference below 0.8m and descriptor L2 distance below a threshold, with the top eight matches retained per scan keypoint.The top-8 selection addresses recurring structures in scans and CAD models.
  • Alignment: RANSAC rejects outliers and estimates translation plus upright rotation from three random correspondences, using a 0.20m inlier threshold and fixed class-average scale.Keypoints used as inliers or lying inside an aligned CAD bounding box are ignored for later registrations.
  • Evaluation: Accuracy is analyzed across translation, rotation, and scale thresholds, with default values of ϵt = 0.2m, ϵr = 20°, and ϵs = 20%.Only one threshold is varied at a time in the sensitivity analysis.
  • Evaluation: Baseline hyperparameters for FPFH, SHOT, and Li et al. are selected through a correspondence benchmark and validation-set search.The search places the baseline methods within a consistent evaluation framework.
Loading 1811.11187v1…