PaperlayerView exact revision on arXiv

Source-linked AI summary

Category-Level 3D Correspondence in Camera Space via Morphable Object Priors

Leonhard Sommer, Artur Jesslen, Basavaraj Sunagad, Adam Kortylewski

arXiv:2605.28257v1cs.CV

TL;DR

Fine-grained semantic understanding of object parts remains limited by pose-centric and image-space representations. The paper introduces HouseCorr3D and Morpheus, using shared morphable shape priors to learn camera-space correspondences without explicit correspondence supervision, and reports state-of-the-art performance on the benchmark.

  • Problem

    Existing 3D object-understanding representations focus on pose, detection, or reconstruction and fail to capture fine-grained semantics for object parts, functions, and interactions.

  • Method

    Morpheus learns morphable category-level shape priors by constraining instances through a shared deformable template with canonical shape and instance-specific deformation.

  • Results

    Morpheus substantially outperforms existing baselines on HouseCorr3D, achieving state-of-the-art camera-space correspondence performance without explicit correspondence supervision.

  • Takeaways & Limitations

    HouseCorr3D enables quantitative evaluation of monocular category-level 3D correspondence with amodal and symmetry-aware annotations, while shared morphable priors provide an effective mechanism for establishing it.

  • Takeaways & Limitations

    A fixed-connectivity shared template cannot handle large topological variation, and camera-space correspondence remains sensitive to pose errors and may oversmooth thin structures.

Abstract

from arXiv · show

Understanding 3D objects from images is fundamental to robotics and AR/VR applications. While recent work has made progress in category-level pose estimation, current representations fail to capture the fine-grained semantics needed for reasoning about object parts, functions, and interactions. In this work, we study category-level 3D correspondence in camera space -- predicting, from a single image, 3D locations that remain consistent across instances within a category -- and show that it can emerge without explicit correspondence supervision by learning a shared morphable object prior. To enable research in this direction, we introduce HouseCorr3D, the first large-scale benchmark for monocular category-level 3D correspondence with 178k images across 50 household object categories, 280 unique instances, and 3D keypoint annotations directly on CAD models. Crucially, HouseCorr3D provides amodal correspondence labels for occluded regions and explicit symmetry annotations, addressing key limitations of existing datasets. We further propose Morpheus, a method that learns morphable category-level shape priors by disentangling canonical shape, deformation, and object pose. Through this shared canonical grounding, semantically meaningful 3D correspondences in camera space emerge implicitly. These emerging 3D correspondences set a new state of the art on HouseCorr3D, demonstrating that semantic 3D object understanding can arise without direct correspondence supervision. Data and code are publicly available at https://github.com/GenIntel/HouseCorr3D.

1 Introduction

The paper defines monocular category-level 3D correspondence as predicting semantically aligned 3D locations across instances, addressing fine-grained understanding beyond pose estimation and 2D matching. It introduces HouseCorr3D and Morpheus, showing that morphable priors can produce correspondence without explicit supervision.

  • Existing 2D correspondences are limited by viewpoint dependence, occlusion, and symmetry ambiguities, motivating correspondence directly in 3D camera space.
  • Monocular category-level 3D correspondence predicts the 3D point in a target image corresponding to a semantic point selected on a query object.
  • HouseCorr3D contains 178k images across 50 household categories and 280 instances, with mesh-based keypoints, amodal correspondences, and explicit symmetry labels.
  • A shared deformable template mediates correspondence and constrains object instances through a common representation.
  • Morpheus learns morphable category-level shape priors to establish semantically consistent 3D correspondences directly in camera space.
  • Morpheus substantially outperforms existing baselines on HouseCorr3D, establishing a new paradigm for correspondence-level 3D object understanding.

2 Related work

Prior work addresses 2D matching, 3D keypoints, canonical mappings, morphable models, and category-level benchmarks, but existing approaches and datasets do not provide monocular category-level 3D correspondence evaluation. HouseCorr3D addresses this gap with semantically consistent CAD-based annotations across household categories.

  • 2D Semantic Correspondence: 2D correspondence methods achieve semantic alignment in image space but do not predict 3D canonical coordinates or enforce semantic consistency across instances in 3D.
  • 3D Keypoint and Correspondence Methods: 3D keypoint and surface-mapping methods establish category-consistent points or canonical coordinates, while mesh reconstruction methods extend correspondence-related modeling.
  • Morphable Models and Shape Priors: Morphable models represent intra-class shape variability through deformable canonical templates, extending from faces and bodies to diverse object classes.
  • Benchmarks for Category-Level 3D Understanding: Existing datasets provide CAD meshes, part labels, pose annotations, or reconstructed-mesh evaluations, but lack consistent point-level correspondences across diverse instances from monocular images.
  • Benchmarks for Category-Level 3D Understanding: HouseCorr3D annotates up to 19 semantically shared 3D keypoints on CAD meshes for 5–13 instances per category across 50 household object classes.

3 The HouseCorr3D Benchmark

HouseCorr3D establishes a benchmark for category-level 3D correspondence in camera space, including amodal and symmetry-aware evaluation across diverse household objects.

  • Benchmark scope: HouseCorr3D targets monocular category-level correspondence in 3D camera space rather than 2D camera or 3D object space.The task predicts a target 3D point corresponding to the same semantic part as a query point from two RGB-D images.
  • Amodal evaluation: Amodal correspondences support evaluation of object parts that are occluded or not visible in the image.This captures the full spatial extent needed for reasoning about grasps and interactions rather than only visible surfaces.
  • Dataset scale: 178k test images and 2.6M training images span 50 categories, with image pairs formed from views containing another instance.The benchmark is built from synthetic Omni6DPose data with CAD models, realistic lighting, clutter, and occlusions.
  • Annotations: The benchmark provides mesh-based keypoints selected for geometric distinctiveness, semantic meaning, and transferability across instances.Annotations cover corners, edges, handle centers, and other salient structural features, using independent annotation and merging procedures.
  • Symmetry handling: HouseCorr3D treats symmetry orbits as valid correspondences, preventing geometrically equivalent predictions from being unfairly penalized.The protocol supports both discrete and continuous symmetries, including rotations around a symmetry axis.

4 Method

Morpheus learns category-level object instances as identity-preserving deformations of a shared template, predicts pose and shape from RGB-D images, and transfers correspondences through persistent mesh identities.

  • Model overview: Morpheus predicts a 6D object pose and a deformable 3D shape from a single RGB-D image.The posed deformed mesh is placed directly in camera space.
  • Correspondence prediction: Correspondence is obtained by projecting a query point onto the query mesh, transferring its barycentric surface identifier, and mapping it onto the target mesh.Shared mesh topology makes the barycentric coordinates transferable between instances.
  • 3D Morphable Priors: The morphable prior represents category instances as deformations of a shared canonical template whose vertex identities persist across instances.Points mapped to the same template vertex correspond to the same semantic part, converting correspondence estimation into pose and deformation estimation.
  • Instance deformation: An image-conditioned latent code drives an affine deformation field that displaces template vertices while preserving mesh connectivity and identities.The deformation encoder uses a DINOv2 backbone and a convolutional head to produce the latent code.
  • Training Objectives: Training uses geometric supervision without explicit correspondence labels, combining amodal mask, distance-transform, 3D reconstruction, and regularization objectives.The optimization proceeds in two stages, first refining the category template with geometric terms and then training the remaining objectives.

5 Experiments

Morpheus is evaluated on HouseCorr3D against 2D and 3D correspondence baselines, achieving state-of-the-art correspondence performance while handling occlusions and real-world transfer.

  • Experimental setup: Morpheus is evaluated on HouseCorr3D against DINOv2, NOCS, MagicPony, and GenPose++ using 2D, 3D modal, and 3D amodal correspondence metrics.The benchmark uses PCK@0.1, with 2D distances measured in pixel space and 3D distances measured in camera space.
  • Main results: Morpheus outperforms all compared 2D and 3D correspondence methods on the reported HouseCorr3D PCK@0.1 evaluations.Table 2 covers 2D, 3D modal, and 3D amodal correspondences on a HouseCorr3D subset.
  • Occlusions: 2.9% average PCK@0.1 drop separates Morpheus’s modal and amodal correspondence performance, while amodal results remain competitive.The qualitative results also show successful reconstruction of occluded parts, unlike the cited 2D feature-matching and MagicPony examples.
  • Normalized object space: Morpheus outperforms NOCS and GenPose++ on modal correspondences, indicating that normalized object space alone is insufficient for the evaluated 3D matching task.GenPose++ can match a query point outside the target object when object size differs.
  • Model comparison: Explicit disentanglement of pose, shape, and canonicalization yields more accurate 3D structure and higher semantic consistency than the entangled MagicPony setup.MagicPony’s deformation can fit 2D images but struggles to recover consistent 3D rotations across views.
  • Benchmark difficulty: 52.7% on SPair71k versus 22.9% on HouseCorr3D shows that the broader category diversity makes HouseCorr3D more challenging for DINOv2.The comparison is reported as a performance gap across the two benchmarks.
  • Real-world transfer: Approximately 7% lower 3D performance on a filtered ROPE subset accompanies roughly comparable 2D performance, with the drop attributed to noisier depth and annotations.The filtered real-world evaluation contains 5 classes, 24 instances, and 134 keypoints.
  • Limitations: A fixed-connectivity shared template cannot handle large topological variation, while pose errors cause global misalignment and smoothness regularization can oversmooth thin structures.Jointly optimizing pose and deformation remains an open problem.

6 Conclusion

The paper introduces category-level 3D correspondence in camera space and evaluates it with HouseCorr3D. Morpheus uses morphable priors to maintain correspondences across varying shapes, poses, occlusions, and symmetries.

  • 6 Conclusion: HouseCorr3D establishes a benchmark for monocular category-level 3D correspondence with mesh-based annotations, explicit symmetries, occlusions, and amodal correspondences.It covers 50 everyday categories in crowded scenes.
  • 6 Conclusion: Morpheus leverages morphable priors and pose- and occlusion-aware supervision to achieve state-of-the-art correspondence performance across instances with varying shapes and poses.The conclusion states that the method morphs objects while maintaining consistent correspondences.
  • 6 Conclusion: The results show that approaches relying only on 2D supervision remain insufficient for category-level 3D correspondence.The benchmark is presented as a foundation for correspondence learning toward embodied robotics applications.

Supplementary Material

The supplementary material expands the paper’s benchmark, experiments, dataset documentation, annotation process, losses, and visualization details.

  • Supplementary Material: The supplementary material provides additional results, experimental setup details, and discussion of limitations in existing benchmarks.It complements the main paper’s quantitative and qualitative analyses.
  • Supplementary Material: Additional sections document HouseCorr3D statistics, the real subset of Omni6DPose, and the mesh annotation process.These sections provide dataset and annotation details beyond the main text.
  • Supplementary Material: The supplement describes auxiliary training losses and introduces the HueGrid visualization.These topics are listed alongside the experimental and dataset supplements.

A Limitation of existing benchmarks

NOCS represents geometric position in a normalized cube rather than semantic part identity, so its correspondence quality degrades as intra-class shape variation increases. Consequently, NOCS coordinates are unreliable as semantic correspondence ground truth.

  • NOCS assigns coordinates from bounding-box geometry without encoding semantic part identity.The same semantic part can occupy different NOCS locations across differently shaped instances.
  • NOCS-based matching can succeed when category shapes are sufficiently rigid but breaks down under high intra-class variation.
  • Using NOCS coordinates for evaluation treats geometric coincidence as semantic alignment, making them an unreliable correspondence proxy.

B Additional results

Additional experiments report the same overall trends as the main paper across the full benchmark, while qualitative failures identify pose, mesh, and depth estimation as error sources. The supplementary material also documents training configurations and baselines.

  • Additional results: The complete per-category results reproduce the main paper’s overall trends across HouseCorr3D.The highlighted categories in the main figures were selected randomly, rather than to favor particular outcomes.
  • Qualitative limitations: Pose errors can produce incorrect projections on the estimated mesh and therefore wrong correspondences.
  • Qualitative limitations: Coarse mesh estimation and incorrect depth estimation can also yield wrong 3D correspondences, even when the 2D projection is accurate.
  • Training setup: Morpheus is trained with multiple components and losses whose hyperparameters, loss weights, and architectures are summarized in the supplementary setup.
  • Baselines: The DINOv2 baseline uses a ViT-S backbone, 448^2 images, a 32^2 patch grid, and L2-normalized feature maps for correspondence computation.
  • Baselines: The NOCS baseline uses a ResNet50 with FPN, rendered canonical XYZ targets, official masks, category labels, and depth.

D Additional dataset statistics

The supplementary dataset statistics explain the choice of synthetic data and document HouseCorr3D’s scale and variation. Annotation coverage differs across categories because instances vary in shape and semantics.

  • Data source: The real Omni6DPose subset offers limited diversity, with most categories containing at most two instances and repeated scenes across video sequences.This restricts the number of reliable correspondences available from real captures.
  • Data source: Synthetic data provides large variation in object instances and scene composition while mimicking real-world lighting and clutter.
  • Dataset scale: HouseCorr3D contains 178k images from 280 unique instances across 50 categories.
  • Annotation coverage: Keypoint coverage varies across classes because object instances can differ in shape and semantics.Toy-plane and toy-train examples include substantially different object types, such as helicopters and bullet trains versus conventional locomotives.

E Real subset of Omni6DPose

The real-data evaluation uses a statistically representative five-class subset and verifies camera-space alignment, while annotation and regularization procedures support semantically consistent correspondences. HueGrid visualizes both absolute correspondences and local deformation.

  • Real subset selection: Five classes—bread, facial cream, hair dryer, handbag, and tooth brush—form the selected real-data subset, with total distance 0.116 to full-dataset statistics.The subset has mean distance 0.109 and variance distance 0.123.
  • Alignment verification: The real subset requires frame-by-frame verification because camera motion around static scenes can produce alignment errors.The evaluation reprojects 3D assets into camera space and checks projected meshes against RGB images.
  • Mesh annotation process: Annotators place up to 20 semantically meaningful keypoints per category on distinctive structures such as corners, handles, wheel centers, and wing tips.
  • Mesh annotation process: Ambiguous keypoints are manually accepted, merged, split, or rejected after visualizing correspondences across instances.The reference mesh is annotated first and subsequent instances are aligned to it using a 3D interface.
  • Additional losses: Additional losses constrain the learned representation to preserve the SDF property, remain near a canonical template, and maintain smooth deformations.The regularizers are intended to suppress spurious local distortions while permitting non-rigid articulation.
  • HueGrid visualization: HueGrid combines 3D-based color coding with checkerboard cues to show absolute correspondence and local geometric deformation simultaneously.

I Discussion about correspondence evaluation

HouseCorr3D evaluates category-level 3D correspondences in camera space using modal and amodal settings while explicitly accounting for discrete and continuous geometric symmetries. Its symmetry-aware metric treats geometrically equivalent predictions as correct and supports standardized evaluation.

  • Modal and amodal evaluation: Modal correspondences evaluate only visible surface points, whereas amodal correspondences extend evaluation to the full object surface, including occluded regions.Amodal regions include self-occlusion, occlusion by another object, and areas outside the camera frustum.
  • Symmetry-aware evaluation: HouseCorr3D explicitly handles discrete and continuous symmetries so geometrically equivalent predictions are not unfairly penalized.The benchmark illustrates valid alternatives for discrete symmetry, such as flipping a pillow, and rotational symmetry, such as rotating around an axis.
  • Symmetry-aware evaluation: For rotational symmetry, the correspondence error is the minimum Euclidean distance over rotations around the object’s fixed symmetry axis.Geometrically, this measures the distance from the prediction to the circular orbit of the ground-truth point.
  • Symmetry-aware evaluation: The symmetry-aware definition counts predictions aligned with any symmetric equivalent point as correct, yielding a fair metric for real-world geometric ambiguities.This design enables robust evaluation of category-level correspondence methods.
  • Evaluation resources: The release includes annotated 3D meshes, projected 2D keypoints, dataset-preparation scripts, and helper functions for computing the proposed correspondence metrics.These resources are intended to support consistent and standardized evaluation.
Loading 2605.28257v1…