PaperlayerView exact revision on arXiv

Source-linked AI summary

SUFLECA: Scaling Up Feature Learning for CAD-to-image Alignment

Saad Ejaz, Miguel Fernandez-Cortizas, Javier Civera, Holger Voos, Jose Luis Sanchez-Lopez

arXiv:2607.15058v1cs.CVcs.RO

TL;DR

CAD-to-image alignment must recover an object's 9D pose from a single RGB image, but existing zero-shot correspondences degrade under occlusion and domain shift. SUFLECA learns geometry-grounded features from diverse real and synthetic data and enforces geometrically consistent matching, achieving 33.4% category and 42.3% instance accuracy on ScanNet25k while surpassing supervised methods.

  • Problem

    Existing CAD-to-image alignment methods struggle to generalize under real-scene clutter, occlusion, appearance variation, and sim-to-real domain shift while relying on costly or noisy matching.

  • Method

    SUFLECA scales NOC-supervised feature learning across diverse real and synthetic datasets and uses mutual, geometrically consistent correspondence estimation for zero-shot 9D alignment.

  • Results

    33.4% category-averaged and 42.3% instance-averaged accuracy on ScanNet25k surpass ZeroCAD by 10.3 and 12.2 percentage points, respectively, while also surpassing fully supervised methods.

  • Takeaways & Limitations

    SUFLECA provides accurate, compact, sub-second zero-shot CAD alignment without pose-annotated training data or iterative refinement.

  • Takeaways & Limitations

    Alignment quality remains limited by CAD retrieval accuracy, and SUFLECA is restricted to indoor scenes and common object categories.

Abstract

from arXiv · show

CAD-to-image alignment aims to estimate an object's 9D pose (rotation, translation, and anisotropic scale) from a single RGB image, enabling applications in robotics and augmented reality. Recent zero-shot methods use visual foundation models to match image regions to CAD models, yet typically their correspondences are appearance-driven and degrade under occlusion or sim-to-real domain shift. To address these limitations, we introduce SUFLECA (Scaling Up Feature LEarning for CAD Alignment), a weakly-supervised framework for zero-shot CAD alignment with two key contributions. First, SUFLECA scales up geometry-grounded feature learning from pretrained visual representations through Normalized Object Coordinates (NOCs) supervision on 674K images spanning 12 real and synthetic datasets, learning compact geometry-aware features that generalize across domains. Second, we propose a geometrically consistent matching algorithm that establishes reliable one-to-one CAD-to-image correspondences. Together, these contributions enable accurate, sub-second alignment per object instance without iterative pose refinement. On ScanNet25k, SUFLECA achieves 33.4%/42.3% category/instance accuracy, outperforming, with a smaller computational footprint, the strongest zero-shot baseline by 10.3/12.2 percentage points and, for the first time on this benchmark, even surpassing fully supervised methods. Code is available at: https://github.com/snt-arg/SUFLECA

I. INTRODUCTION

SUFLECA addresses the domain gap and efficiency limitations of zero-shot CAD-to-image alignment by scaling NOC-supervised feature learning and enforcing geometrically consistent correspondences. Together, these contributions improve generalization while enabling accurate alignment with sub-second runtime and reduced matching cost and memory.

  • I. INTRODUCTION: 9D alignment estimates rotation, translation, and anisotropic scale after retrieving an approximately matching CAD model for the observed object.The alignment step is typically solved by establishing CAD-to-image correspondences.
  • I. INTRODUCTION: Foundation-model features improve zero-shot generalization but remain appearance-based, lack 3D geometry grounding, and cause a domain gap that degrades alignment accuracy.Earlier supervised approaches also require costly pose-annotated data and generalize poorly to out-of-distribution images.
  • I. INTRODUCTION: Synthetic NOC supervision alone struggles with clutter, occlusion, and appearance variation, while high-dimensional nearest-neighbor matching is costly and fails to exploit geometric consistency.These limitations affect both the learned feature space and the correspondence-estimation procedure.
  • I. INTRODUCTION: SUFLECA scales NOC-supervised feature learning across a large mixture of real and synthetic datasets to improve generalization and produce a low-dimensional feature space.The resulting representation reduces matching cost and memory footprint, supporting robotic applications.
  • I. INTRODUCTION: The proposed correspondence algorithm enforces mutuality and geometric consistency instead of relying on conventional semantic-only nearest-neighbor matching.Together with scaled feature learning, this enables zero-shot alignment that surpasses supervised methods while maintaining sub-second runtime.

II. RELATED WORK … C. Zero-shot CAD-to-image alignment

Prior work frames CAD-to-image alignment through NOC-based coordinate prediction, supervised pose-estimation paradigms, and zero-shot methods that reduce annotation dependence but retain computational, prior, or robustness limitations. The related work also motivates geometry-grounded supervision and correspondence filtering for more generalizable alignment.

  • A. Normalized Object Coordinates (NOCs): NOCs encode object points in a category-level canonical frame, enabling pose estimation through dense coordinate prediction followed by 3D registration.OmniNOCS introduced large-scale NOC supervision, while vision backbones’ implicit 3D awareness motivates NOC supervision over frozen features.
  • B. Supervised CAD-to-image alignment: Supervised CAD alignment methods use direct 9D pose regression, render-and-compare refinement, or other feed-forward and optimization-based paradigms.Examples include Total3DUnderstanding, Mask2CAD, and 3D-RCNN.
  • C. Zero-shot CAD-to-image alignment: DiffCAD and SDFit reduce dependence on pose-annotated data but require per-category priors and are computationally expensive.DiffCAD estimates scale, pose, and shape with cascaded diffusion models, whereas SDFit fits a morphable SDF through iterative render-and-compare optimization.
  • C. Zero-shot CAD-to-image alignment: The validation set combines RGB images, ground-truth NOC maps overlaid on RGB, and synthetic counterparts generated from CAD annotations.The figure describes example frames from a multi-dataset validation set.
  • C. Zero-shot CAD-to-image alignment: Related category-level methods progressively prune outliers to recover the largest compatible measurement set, but depend on active shape models and keypoint detectors.These dependencies are difficult to obtain for most categories, motivating alternative correspondence filtering.
  • C. Zero-shot CAD-to-image alignment: Diorama uses an additional off-the-shelf model for scale inference and targets open-world scene-level layout, but its accuracy degrades in real-world scenes with occlusions.The passage also introduces ZeroCAD as a recent NOC-supervised method, but the supplied text ends before describing it further.

III. SUFLECA · A. Problem Definition · B. Overall CAD Fitting Pipeline

SUFLECA estimates a 9D CAD-to-camera transformation and an alignment-quality score from an RGB image, camera intrinsics, and a retrieved CAD candidate. Its indirect pipeline extracts masks and descriptors, establishes image-to-CAD correspondences, lifts them to 3D, and solves alignment geometrically while improving feature learning and correspondence consistency.

  • A. Problem Definition: The task estimates a 9D transformation from the CAD coordinate frame to the camera coordinate frame using an RGB image, camera intrinsics, and a retrieved CAD model.The transformation is defined between Xcad and Xcam.
  • A. Problem Definition: The transformation includes rotation R ∈SO(3), axis-aligned anisotropic scale S = diag(sx, sy, sz), and translation t ∈R3.SUFLECA also outputs Sfit ∈R to measure estimated alignment quality for ranking or multi-view aggregation.
  • B. Overall CAD Fitting Pipeline: The indirect CAD alignment pipeline extracts an object mask and per-pixel descriptors from the input image, while also extracting descriptors for the candidate CAD model.This follows the standard pipeline used in prior work.
  • B. Overall CAD Fitting Pipeline: The feature architecture fuses multi-scale features from a frozen perception encoder, upsamples them with a Dense Prediction Transformer, and decodes them through a lightweight binned NOC head.The design aligns DPT features to NOC space while preserving discriminative capacity for correspondence estimation.
  • B. Overall CAD Fitting Pipeline: ScanNet is omitted during training when evaluating on ScanNet25K, producing a strict zero-shot setting with 557,567 frames and 3.55 annotated objects per frame.The datasets above the table’s horizontal line are real-world, and those below are synthetic.
  • B. Overall CAD Fitting Pipeline: Image-to-CAD correspondences are established in feature space, lifted to 3D with monocular depth estimates, and passed to a RANSAC-based Procrustes solver.The solver estimates the alignment specified in Equation (1).
  • B. Overall CAD Fitting Pipeline: SUFLECA improves the pipeline through a feature model learned at scale across real and synthetic data and a correspondence module that enforces geometric consistency.These are the two key modules improved by SUFLECA.

C. Scaling Unified NOC-aligned Feature Learning

SUFLECA scales geometry-aware feature learning by pairing real images with pose-matched synthetic CAD renders and supervising dense features with derived NOC maps. A lightweight NOC head trains the representation, which is discarded at test time while normalized features support correspondence estimation.

  • Multi-Dataset Setup: The training setup augments annotated real images with pose-matched synthetic CAD renders using randomized backgrounds, textures, and occasional visually similar object replacements.This produces a larger, more balanced real-synthetic dataset with comparable occlusion patterns and scene context.
  • NOC Derivation and Verification: NOC maps are derived by rendering the ground-truth object, extracting a SAM2 mask, backprojecting the region into 3D, and computing NOCs from depth.Depth comes from available depth maps or a monocular metric depth estimator.
  • Model Architecture: A frozen pretrained perception encoder feeds patch features to a Dense Prediction Transformer that fuses multi-scale outputs into dense feature maps.The resulting DPT features have dimension dim and are decoded by the NOC head.
  • Model Architecture: The lightweight NOC head bins each NOC coordinate axis into m uniform bins and estimates coordinates from bin-probability expectations, encouraging smooth geometric features.Training includes a categorical cross-entropy term between predicted and ground-truth NOC values.
  • Model Architecture: At test time, SUFLECA discards the NOC head and uses ℓ2-normalized DPT features as correspondence descriptors, optionally concatenated with backbone output features.The optional concatenation is intended to mitigate forgetting from weak supervision over limited category sets.

D. Geometrically-consistent Correspondence Estimation

SUFLECA estimates CAD-to-image correspondences entirely in 2D, combining mutual k-nearest-neighbor matching with scale-normalized geometric consistency and robust inlier selection. It then recovers the 9D alignment using anisotropic Procrustes registration and evaluates quality from registration residuals.

  • 2D Feature Matching: SUFLECA keeps matching entirely in the 2D domain, using image features and a small set of pre-rendered CAD views with precomputed features.This avoids aggregating 2D features across multiple renders into lossy, arbitrary mesh-level 3D features.
  • Correspondence Filtering: Mutual k-nearest-neighbor matching retains reciprocal descriptor matches and resolves duplicates through greedy assignment to the highest-scoring source match.This improves on nearest-neighbor matching, which can produce geometrically inconsistent and many-to-one correspondences.
  • Geometric Consistency: An approximate isotropic scale initializes anisotropic log-scales with IRLS, followed by shrinkage toward the isotropic estimate when axis coverage is insufficient.The isotropic estimate is obtained from the histogram mode of per-pair isotropic log-scales.
  • Geometric Consistency: A consensus matrix and principal eigenvector score geometric consistency, after which gated inliers are passed to space-partitioning RANSAC for anisotropic Procrustes registration.The method uses pairwise distance preservation after scale normalization, rather than applying it directly under anisotropic scaling.
  • Alignment Quality: Sfit = log det(Λ) measures alignment quality from an analytically computed registration information matrix based on residual Jacobians.Unlike CAD retrieval scores used by prior works, Sfit is derived from registration residuals and the 9D pose parameterization.

IV. EVALUATION · A. Implementation Details

SUFLECA uses frozen DUNE encoders to produce compact patch features for geometry-grounded NOC prediction. Training uses a shallow two-layer convolutional NOC head with discretized coordinate classification, while alignment maintains six pre-rendered fixed views.

  • A. Implementation Details: SUFLECA uses frozen DUNE-B to produce DPT patch features with dim = 384 for NOC prediction.The main feature model uses DUNE-B as its frozen perception encoder.
  • A. Implementation Details: The NOC head has two shallow convolutional layers and classifies each coordinate into m = 64 bins.This discretizes normalized object-coordinate supervision into 64 classes per coordinate.
  • A. Implementation Details: Training runs for 50 epochs with AdamW, λ = 0.33, and a learning rate of 1.1 × 10−4.These are the reported optimization settings for the main model.
  • A. Implementation Details: SUFLECA-S uses DUNE-S with dim = 256 and m = 50 as a smaller model variant.The smaller variant reduces both feature dimensionality and NOC-bin count relative to the main configuration.
  • A. Implementation Details: For alignment, SUFLECA maintains n = 6 pre-rendered fixed views.The passage specifies six fixed views for the alignment procedure.

B. Methodology

SUFLECA is evaluated across complementary settings, using ScanNet25k as the main alignment-accuracy benchmark and CO3D to assess generalization to unseen categories. The evaluation also includes ablations, robustness analysis for inexact CAD retrieval, and efficiency analysis.

  • Evaluation Settings: ScanNet25k serves as the main benchmark for alignment accuracy.
  • Evaluation Settings: CO3D is adapted to assess generalization to categories not observed during weak supervision.
  • Evaluation Settings: Ablation studies isolate the contributions of SUFLECA’s design choices.
  • Evaluation Settings: Additional analyses examine robustness to inexactly retrieved CAD models and computational efficiency.

C. Datasets · D. Results

ScanNet25k evaluates zero-shot CAD alignment under strict translation, rotation, and scale thresholds using category- and instance-averaged accuracy. SUFLECA outperforms zero-shot baselines on ScanNet25k and competing methods under challenging CO3D occlusion and CAD-retrieval conditions.

  • C. Datasets: ScanNet25k considers an alignment correct when translation, rotation, and scale errors are below 20 cm, 20°, and 20%, respectively.Accuracy is reported as category- and instance-averaged performance over nine categories on the validation split.
  • C. Datasets: The ScanNet25k evaluation ranks detections by the product of ROCA semantic retrieval score and Sfit, following an NMS-based protocol.Methods use ROCA boxes and CAD retrievals, SAM2 masks, and monocular metric depth; ScanNet-derived images are excluded from weak supervision for zero-shot evaluation.
  • D. Results: 33.4% category-averaged and 42.3% instance-averaged accuracy are achieved by SUFLECA on ScanNet25k, surpassing ZeroCAD by 10.3 and 12.2 percentage points.These results use the NMS protocol and outperform FoundationPose by a larger margin.
  • D. Results: SUFLECA’s ScanNet25k gains are attributed to broader, more diverse NOC-supervised training data and geometrically reliable correspondences.The training data reduce the domain gap to real images; the supplied passage truncates the correspondence explanation.
  • D. Results: ScanNet25k and DiffCAD are evaluated at 20 cm/20°/20% thresholds with category- and instance-average accuracies, distinguishing full, weak, and unsupervised supervision.Both weakly supervised and unsupervised methods operate in the zero-shot setting.
  • D. Results: SUFLECA-blend performs on par with or slightly below SUFLECA because DUNE-B features primarily improve generalization, which is less critical for common ScanNet25k categories.The comparison concerns the feature space and common object categories in ScanNet25k.
  • D. Results: On CO3D, occluded masks and inexact retrieved CADs create substantial front–back and left–right geometric ambiguities, yet SUFLECA grounds objects in 3D better than competing methods.CO3D results use selected sequences; Table III reports category-median aggregation across 600 images and defines the reported metrics and units.
  • D. Results: In seen CO3D categories, all SUFLECA variants perform comparably and outperform competing methods, highlighting the effectiveness of weak supervision.Seen categories include chair and couch; unseen categories include toaster, hairdryer, microwave, and suitcase.

E. Ablation Study · F. Robustness to Inexact CADs

The ablations show that additional in-distribution data, mutual matching, geometric consensus, and the combined NMS score each improve ScanNet25k alignment. Retrieval quality remains a major bottleneck, but SUFLECA is moderately robust when retrieved CAD models are inexact.

  • E. Ablation Study: 34.4/42.8 alignment accuracy is achieved when ScanNet-derived images are added to training, up from 33.4/42.3.The gain is 1.0/0.5 percentage points, indicating additional data helps while strong features are learned without in-distribution images.
  • E. Ablation Study: 32.0/41.0 accuracy without the full geometric-consistency pipeline shows that mutual matching and geometric consensus are both important.The reduced variant still underperforms the full pipeline.
  • E. Ablation Study: 30.4/39.3 accuracy results from replacing the combined Sfit-based NMS score with the ROCA retrieval score, versus 33.4/42.3.The 3.0/3.0 percentage-point drop shows retrieval confidence alone is insufficient for ranking alignments.
  • F. Robustness to Inexact CADs: More than 20 percentage points separate ground-truth Scan2CAD annotations from GroundedSAM w/ OSCAR zero-shot retrieval in category- and instance-averaged alignment accuracy.The gap is primarily attributed to low instance retrieval accuracy under inexact CAD retrieval.
  • F. Robustness to Inexact CADs: 3.8% instance retrieval accuracy for GroundedSAM w/ OSCAR makes most retrieved CADs inexact.Heavy occlusion, appearance differences, and an additional reported factor contribute to the low retrieval accuracy.
  • F. Robustness to Inexact CADs: SUFLECA is moderately robust to CAD inexactness because instance retrieval accuracy drops more steeply than alignment accuracy from oracle to supervised ROCA to fully zero-shot retrieval.Zero-shot CAD retrieval from real-world observations remains an open problem and a key bottleneck for fully zero-shot CAD fitting.

G. Efficiency Analysis · V. CONCLUSION

SUFLECA reduces alignment resource demands through compact, geometry-aware features and provides sub-second, refinement-free zero-shot 9D CAD alignment. Its NOC-supervised learning and geometrically consistent matching address domain gaps and spatial inconsistencies, yielding state-of-the-art ScanNet25K performance that surpasses supervised methods.

  • G. Efficiency Analysis: SUFLECA achieves lower runtime and VRAM usage than competing methods through compact feature representations.The efficiency comparison excludes detection, segmentation, depth estimation, and retrieval because these stages are consistent across methods.
  • G. Efficiency Analysis: ZeroCAD is evaluated using ZeroCAD*, a runtime-only proxy implementation that estimates a lower bound on its true runtime.ZeroCAD is not open-source, so the comparison uses a proxy rather than the original implementation.
  • V. CONCLUSION: SUFLECA is a weakly-supervised zero-shot method for 9D CAD alignment from images.The method combines scaled NOC-supervised feature learning across diverse real and synthetic data with a mutual k-nearest-neighbors matching algorithm.
  • V. CONCLUSION: SUFLECA’s scaled NOC-supervised feature learning uses diverse real and synthetic data to address the domain gap.The conclusion identifies this learning strategy as one of the method’s two core contributions.
  • V. CONCLUSION: Its mutual k-nearest-neighbors matching algorithm with geometric consensus filtering addresses spatial inconsistencies limiting prior approaches.This is the second core contribution identified in the conclusion.
  • V. CONCLUSION: SUFLECA enables sub-second CAD alignment without pose-annotated training data or iterative refinement.It uses compact, geometrically discriminative features to achieve this capability.
  • V. CONCLUSION: SUFLECA achieves state-of-the-art performance on ScanNet25K and surpasses supervised methods on this benchmark for the first time.The conclusion presents this as an outcome of jointly addressing domain gaps and spatial inconsistencies.
  • V. CONCLUSION: The conclusion acknowledges that important limitations remain after SUFLECA’s reported advances.The supplied passage begins listing limitations but does not provide their full details.
Loading 2607.15058v1…