PaperlayerView exact revision on arXiv

Source-linked AI summary

GenCOPE: Syn2Real Generalized Category-Level Object Pose Estimation for Robotic Picking

Jian Liu, Wei Sun, Zhenqi Dai, Hui Yang, Jian Xiao, Nicu Sebe, Na Zhao

arXiv:2610.01758v1cs.CV

TL;DR

Category-level pose estimation generalizes to unfamiliar instances, but collecting real training data for each novel category remains labor-intensive. GenCOPE trains on synthetic data and uses semantic consistency across 2D and 3D features with cross-modal pose learning; the paper reports strong domain generalization with low computational overhead and real-time inference.

  • Problem

    COPE generalizes to unknown instances within a category, but methods still require labor-intensive real-world training data for novel categories, limiting scalability.

    Source: §1 · p. 2

  • Method

    GenCOPE uses LLM-generated 2D and 3D semantic descriptions encoded by CLIP to optimize global features, then applies cross-modal learning for pose regression.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    GenCOPE achieves superior domain generalization with minimal computational overhead and real-time inference.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The authors conclude that GenCOPE is well-suited for practical integration into real-world robotic platforms.

    Source: §5 · p. 10

    The current formulation does not explicitly account for pose equivalence in strongly symmetric categories.

    Source: §5 · p. 10

Abstract

from arXiv · show

Category-level object pose estimation (COPE), capable of generalizing to intra-class unknown objects, has become a core technique for robotic 3D scene understanding. However, existing COPE methods still require labor-intensive recollection of real-world training data for novel object categories, which limits their scalability in practical applications. This paper aims to achieve synthetic-to-real (Syn2Real) generalized COPE, where a model is trained solely on rendered synthetic data and directly generalized to real-world deployments. The central challenge lies in the significant domain gap between synthetic and real-world data, particularly in texture appearance. To address this, we aim to enhance domain generalization by learning domain-invariant representations that capture semantic commonalities among objects within the same category. We introduce 2D and 3D semantic consistency constraints to reduce the sensitivity of feature encoders to domain-specific features. In addition, we propose an end-to-end pose regression framework that performs 2D-3D cross consistency learning, leveraging dense cross-modality fusion to further refine pose estimation. Since simplicity and effectiveness are essential for real-world robotic deployment, our model operates exclusively on global features, yielding a highly lightweight and efficient architecture. Extensive experiments on the REAL275 and Wild6D benchmarks, as well as real-world robotic manipulation scenes, show superior Syn2Real generalization performance of our paradigm. Code and demos are released at https://paperreview99.github.io/GenCOPE/.

1 Introduction

GenCOPE targets synthetic-to-real category-level pose estimation by addressing texture-driven domain differences with semantic constraints on 2D and 3D representations. Its lightweight global-feature pipeline combines cross-modal learning with end-to-end 9DoF pose regression.

1 Introduction

  • COPE estimates poses of unknown instances within a category, while 9DoF pose includes an object's translation, rotation, and size in camera coordinates.The paper positions COPE as a direction beyond instance-level methods restricted to training-seen objects.Source: §1 · p. 2
  • Existing COPE methods require labor-intensive real-world training-data collection for each novel category, motivating GenCOPE's synthetic-only training and direct real-world deployment.The paper argues that this requirement limits scalability and practicality in large-scale robotic applications.Source: §1 · p. 2
  • Texture appearance is identified as the most pronounced synthetic-to-real discrepancy, while fine-grained feature extractors encode texture and miss domain-invariant properties.The paper links this texture sensitivity to representations that are suboptimal for robust Syn2Real pose estimation.Source: §1 · p. 2
  • Because intra-class objects retain largely consistent 2D semantic structures and 3D geometry across domains, GenCOPE constrains feature encoding to emphasize those shared cues.The paper illustrates this with bottle parts and corresponding geometric components.Source: §1 · p. 2
  • GenCOPE aligns 2D and 3D features with semantic supervision, fuses them through cross-consistency learning, and regresses full 9DoF pose using global features from lightweight encoders.The framework uses ResNet18 and PointNet encoders and a pose-decouple decoder.Source: §1 · p. 2

2 Related Work

Prior COPE work includes shape-prior methods and prior-free alternatives, while label-efficient approaches reduce reliance on real-world annotations through varied learning strategies. GenCOPE distinguishes itself by explicitly constraining 2D and 3D encoders with semantic knowledge shared across synthetic and real domains.

Fully-Supervised COPE Methods

  • Shape-prior methods such as SPD [41] deform category-level CAD libraries to reconstruct NOCS representations, but manually building those libraries requires substantial human effort.The paper notes that NOCS [2] established two widely adopted COPE benchmarks.Source: §2 · p. 3
  • Prior-free and RGB-D methods pursue varied pose-estimation strategies, but fully supervised approaches still depend on extensive real-world annotation for novel categories.Examples include FS-Net [47], GPV-Pose [48], and RGB-D approaches [34] [53].Source: §2 · p. 3

Label-Efficient COPE Methods

  • Label-efficient COPE spans reduced-supervision voting and shape alignment, domain generalization, adaptation, and diffusion-based Syn2Real methods.Examples include CPPF [57], SAR-Net [58], DPDN [33], UDA-COPE [61], TTA-COPE [62], DiffusionNOCS, and Diff9D [65].Source: §2 · p. 3

2 Related Work

  • Unlike the described prior methods, GenCOPE explicitly constrains both 2D and 3D feature encoding with semantic knowledge to focus on properties shared across synthetic and real domains.The paper presents this design as a way to alleviate adverse effects from domain-specific information.Source: §2 · p. 3

3 Workflow of GenCOPE

GenCOPE targets 9DoF pose estimation for intra-class unknown real-world objects using models trained only on synthetic data. Its workflow combines semantic consistency constraints, 2D–3D feature fusion, and decoupled pose regression.

3 Workflow of GenCOPE

  • GenCOPE predicts translation, rotation, and size for intra-class unknown real-world objects using a model trained only on synthetic data.Source: §3 · p. 3

3.1 Overview

  • The workflow combines 2D and 3D semantic encoders with cross-modal fusion before a decoder regresses the full 9DoF object pose.Source: Figure 2 · p. 4, §3.1 · p. 4

3.2 2D and 3D Semantic Consistency Encoders

  • Gemini-3-generated class descriptions, encoded with CLIP, guide 2D and 3D features toward intra-class semantic invariants and away from domain-specific cues.Ten distinct descriptions are generated per class, with one randomly selected for each training sample.Source: §3.2 · p. 4

3.3 2D-3D Cross Consistency Learning for Pose Regression

  • The cross-consistency framework fuses semantically consistent 2D and 3D features through stacked cross transformers and supports end-to-end pose decoding.Source: §3.3 · p. 5

3.3.1 2D-3D Cross Consistency Learning

  • Each of 2N blocks applies modality-specific self-attention, then cross-attention exchanges information using same-modality queries and opposite-modality keys and values.Symmetric residual pairing combines shallow and deep semantic representations.Source: §3.3.1 · p. 5

3.3.2 End-to-End Pose Decouple Decoder

  • The outermost symmetric pair forms Fout, which feeds separate MLPs that regress translation, rotation, and size.Source: §3.3.2 · pp. 5–6

3.4 Training and Inference Protocol

  • Training jointly optimizes semantic-consistency and pose losses; inference removes textual supervision and predicts 9DoF pose in one RGB-D forward pass without text or extra optimization.Source: §3.4 · p. 6

4 Experiments

Experiments train GenCOPE on synthetic CAMERA25 and evaluate it on REAL275, Wild6D, and robotic picking scenes. The reported comparisons show strong performance across the benchmarks, efficient inference, and benefits from both semantic consistency and 2D–3D cross-modal learning.

4 Experiments

  • Experiments train on synthetic CAMERA25, then evaluate on real-world REAL275, Wild6D, and self-collected robotic picking scenes.The evaluation reports pose accuracy and inference memory and speed; REAL275 testing uses two segmentation protocols.Source: §4 · p. 6, §4.1 · p. 7

4.1 Comparisons with State-of-the-Art Methods

  • GenCOPE achieves the best overall IoU3D and n◦mcm performance on REAL275 and Wild6D in the reported comparisons.On REAL275, it outperforms methods across both segmentation protocols; on Wild6D, it surpasses supervised approaches and Diff9D under the same domain-generalized setting.Source: Table 1 · p. 7, §4.1 · pp. 7–8, Table 2 · p. 8
  • GenCOPE uses the least inference resources and runs fastest among CleanPose and Diff9D on the same computing platform.The authors attribute this efficiency to lightweight encoders, global features in cross-consistency learning, and a compact pose-decoupled decoder.Source: §4.1 · pp. 8–9, Table 3 · p. 8
  • In qualitative comparisons, GenCOPE produces tighter REAL275 bounding boxes than CleanPose and more stable Wild6D orientations than Diff9D.In robotic scenes, it gives more reliable predictions than CleanPose and Diff9D, though severe occlusion, transparent objects with incomplete depth, and extreme low light remain failure cases.Source: Figure 3 · p. 8, §4.1 · p. 9, Figure 4 · p. 9

4.2 Ablation Studies

  • Removing semantic consistency supervision degrades performance, with removing 2D consistency causing a larger reported regression than removing 3D consistency.Replacing part-structured descriptions with class names also reduces performance; changing the LLM while keeping other settings fixed gives comparable accuracy.Source: §4.2 · p. 10
  • Removing 2D–3D cross-consistency learning or either attention component degrades accuracy relative to the full model.The ablations compare cross-modal learning with direct feature concatenation and separately remove self- or cross-attention.Source: §4.2 · p. 10

5 Conclusion

GenCOPE trains on synthetic data and targets robust real-world generalization through semantic consistency-guided 2D/3D encoders and cross-consistency pose decoding, with low computational overhead and real-time inference. Its current formulation does not explicitly model pose equivalence in strongly symmetric categories.

5 Conclusion

  • GenCOPE achieves superior domain generalization with minimal computational overhead and real-time inference for practical robotic-platform integration.The paper presents it as suitable for real-world robotic deployment.Source: §5 · p. 10
  • GenCOPE combines semantic consistency-guided 2D and 3D encoders with 2D-3D cross-consistency learning and an end-to-end pose-decoupled decoder.The method is designed to generalize from synthetic-only training to real-world deployment.Source: §5 · p. 10
  • Strongly symmetric categories remain a limitation because the formulation does not explicitly account for equivalent rotations describing the same physical configuration.The authors identify symmetry-aware losses and evaluation protocols as future work.Source: §5 · p. 10

A Additional Semantic Samples Generated by LLM

The appendix illustrates semantic descriptions generated for object categories and explains how varied language-model outputs are sampled as training supervision.

A Additional Semantic Samples Generated by LLM

  • Ten language-model queries per category yield varied semantic descriptions, with one description randomly sampled at each training iteration.The authors use this variability to increase the diversity and stochasticity of semantic supervision.Source: A Additional Semantic Samples… · p. 16
  • The query requests the common 2D/3D semantic information for a class of objects enclosed in parentheses, using a template that lists the class’s number of features and each feature.Source: A Additional Semantic Samples… · p. 16
  • Bottle descriptions list a cap, neck, and base container in 2D, and a neck cylinder, transition cone, and bottom cylinder in 3D.These samples illustrate category semantics expressed as image parts and geometric components.Source: A Additional Semantic Samples… · p. 16
  • Bowl descriptions use a rim and basin in 2D versus a rotationally symmetric curve and cavity in 3D.The appendix also shows diverse samples for different object categories.Source: A Additional Semantic Samples… · p. 16

B GenCOPE Guided Robotic Picking

GenCOPE runs on embedded hardware in a real-world eye-in-hand robotic picking setup and is used to pick multiple objects sequentially from estimated poses.

B GenCOPE Guided Robotic Picking

  • The system runs entirely on the onboard NVIDIA GTX 1660 GPU with 6 GB memory, without offloading computation.The eye-in-hand platform uses an Intel RealSense L515 RGB-D camera.Source: B GenCOPE Guided Robotic Picking · p. 16
  • Real-world trials sequentially pick multiple scene objects, including textureless metallic components, using estimated center depths.The authors report these experiments as demonstrating deployability, efficiency, and robust Syn2Real generalization; Fig. B visualizes continuous multi-object picking.Source: B GenCOPE Guided Robotic Picking · p. 17
  • For safe grasp execution, the grasp position is manually offset 2 cm above the tabletop and orientation tilted upward by 30°.Objects are then picked sequentially based on their estimated center depths.Source: B GenCOPE Guided Robotic Picking · p. 17

C Details of Some Modules

The module details specify the feature encoders for RGB images and back-projected point clouds, and the shared transformer block used for attention.

C Details of Some Modules

  • ResNet18 without its final fully connected layer extracts 2D features from the RGB image, while a PointNet-based encoder processes the back-projected point cloud.Figure C.1 details the PointNet-based global feature encoder.Source: C Details of Some Modules · p. 17, Figure C.1 · p. 17
  • The PointNet-based encoder is used exclusively for global feature extraction to prioritize deployability and computational efficiency.The point cloud is back-projected from the depth map.Source: C Details of Some Modules · p. 17
  • A shared transformer block implements both self-attention and cross-attention in the 2D-3D cross-consistency network.Figure C.2 identifies residual connection and matrix multiplication symbols in the block.Source: C Details of Some Modules · p. 17, Figure C.2 · p. 18

D Additional Ablation Studies

The REAL275 ablation tests the effects of CLIP text-feature fusion and 2D/3D semantic consistency supervision. Its reported pattern favors consistency supervision over direct feature fusion.

D Additional Ablation Studies

  • The REAL275 ablation varies CLIP text-feature use during training and inference and whether 2D/3D semantic consistency supervision is enabled.Text features are introduced by residual addition or feature concatenation.Source: D Additional Ablation Studies · p. 18
  • Feature concatenation performs comparably to the full model, whereas residual addition slightly degrades performance.Directly injecting text features does not consistently improve performance.Source: D Additional Ablation Studies · pp. 18–19
  • Semantic consistency supervision consistently improves performance with or without explicit text-feature fusion, favoring semantic alignment over direct multimodal fusion.The authors report that this supports focusing the encoders on domain-invariant semantic cues.Source: D Additional Ablation Studies · p. 19

Related papers

Loading 2610.01758v1…