PaperlayerView exact revision on arXiv

Source-linked AI summary

F2G-Pose: Geometry-Aware Foundation Feature Lifting for Direct RGB-D Category-Level Object Pose Estimation

Jinyu Zhang, Haitao Lin, Jiashu Hou, Xiangyang Xue, Yanwei Fu

arXiv:2610.31623v1cs.CV

TL;DR

Category-level RGB-D pose estimation must recover pose from observations made ambiguous by symmetry, occlusion, and variation, while existing approaches face reliability or inference-efficiency challenges. F2G-Pose jointly predicts pose, size, and completed camera-frame geometry in a single pass; it reports state-of-the-art accuracy at 28 FPS and demonstrates manipulation potential with segmented RGB-D inputs.

  • Problem

    Category-level RGB-D pose estimation must recover rotation and translation without instance-specific CAD models or reference views despite ambiguity from symmetry, occlusion, and intra-class variation.

    Source: §1 · p. 1

  • Method

    F2G-Pose lifts foundation visual features to partial point clouds and uses a MoE Transformer to jointly predict pose, metric size, and camera-frame completed geometry in one feed-forward pass.

    Source: §1 · p. 2

  • Results

    F2G-Pose achieves state-of-the-art accuracy while running at 28 FPS for pose inference, with evaluations across synthetic benchmarks and real-world datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Manipulation experiments demonstrate deployment potential under segmented RGB-D inputs, while the shape branch provides dense geometric supervision and approximate planning geometry.

    Source: §1 · p. 2, §5 · p. 10

    F2G-Pose assumes segmented RGB-D crops, remains sensitive to severe occlusion and symmetric low-texture objects, and has weak high-precision metric alignment under cross-dataset domain shift.

    Source: §5 · p. 10

Abstract

from arXiv · show

This paper studies category-level RGB-D object pose estimation, which recovers an object's 3D rotation and translation without instance-specific CAD models or reference views. This task is highly ambiguous due to intra-class variations, symmetries, and partial observations. Current solutions struggle to balance reliability and efficiency: methods based on intermediate structures are prone to error propagation, while diffusion models suffer from high inference latency and complex post-hoc candidate extraction. To address this, we present F2G-Pose, a real-time direct, single-pass framework that predicts category-level pose, metric size, and camera-frame completed shape from a single RGB-D observation in one feed-forward pass. F2G-Pose lifts dense visual foundation features to partial point clouds, encodes local geometry with DGCNN tokenization, and integrates object-level structure with a Mixture-of-Experts Transformer. Joint size prediction and camera-frame shape completion encourage metric consistency through dense geometric supervision. With its task-specific network trained only on pose-annotated synthetic SOPE data and its frozen RADIO backbone externally pretrained, F2G-Pose achieves state-of-the-art accuracy on SOPE and ROPE while running at 28 FPS for pose-only inference, and further demonstrates strong camera-frame geometry on HouseCat6D and real-object transfer on HANDAL. Analyses of latency, robustness, shape completion, and GenPose++ candidate behavior further demonstrate the efficiency and reliability of F2G-Pose, with real-world manipulation experiments demonstrating deployment potential under segmented RGB-D inputs. The project page, code, and pretrained models are made available at https://jrryzh.github.io/F2G-Pose/.

1 Introduction

F2G-Pose targets ambiguous category-level RGB-D pose estimation with a direct, single-pass prediction of pose, size, and completed geometry. Its evaluations report state-of-the-art accuracy at real-time speed, alongside manipulation experiments.

1 Introduction

  • Single-view category-level estimation is challenging because symmetry, occlusion, and intra-class variation leave incomplete evidence for reliable metric predictions.The practical estimates are coupled with metric size and usable geometry for evaluation, physical interaction, and robotic control [37] [14] [15] [25] [44].Source: §1 · p. 1
  • Deterministic methods use intermediate structures that can compound errors when occlusion, missing depth, intra-class variation, or domain shift undermines their reliability.Direct regression avoids some post-hoc alignment, but still needs representations that retain local geometry and global structure under partial observations.Source: §1 · p. 2
  • Sampling-based methods model pose ambiguity, but their deployed accuracy and practical cost also depend on post-hoc candidate extraction and iterative inference.GenPose [50] and GenPose++ [51] sample hypotheses for scoring, filtering, clustering, or aggregation; Joint-RegDiff [20] and RFMPose also use iterative generative inference.Source: §1 · p. 2
  • The framework jointly predicts category-level pose, metric size, and camera-frame completed geometry in one feed-forward pass.It lifts dense visual foundation features to partial point clouds, encodes local geometry with DGCNN, and integrates object-level structure with a Mixture-of-Experts Transformer.Source: §1 · p. 2
  • F2G-Pose achieves state-of-the-art accuracy while running at 28 FPS for pose inference, with robotic experiments demonstrating deployment potential under segmented RGB-D inputs.The task-specific network is trained on pose-annotated synthetic data from 149 SOPE categories and evaluated on SOPE, ROPE, HANDAL, and HouseCat6D.Source: §1 · p. 2

2 Related work

Related work covers category-level methods that use canonical representations, geometry, shape modeling, or generative pose inference, as well as instance-specific reconstruction and alignment. F2G-Pose addresses the category-level setting with camera-frame shape prediction and MoE encoding.

2 Related work

  • Category-level approaches span canonical-space alignment without instance-specific CAD models, geometry and symmetry-aware losses [50], and shape-aware modeling.Many rely on intermediate coordinates, correspondences, deformation fields, or shape priors that can degrade under occlusion, missing depth, and strong intra-category variation.Source: §2 · pp. 2–3
  • GenPose [50] and GenPose++ [51] model multimodal pose and size distributions, while Joint-RegDiff [20] and RFMPose retain iterative generative inference and its associated cost.Joint-RegDiff [20] uses time-dependent score scaling, and RFMPose uses Riemannian flow matching; both remain limited to 6D pose estimation.Source: §2 · p. 3
  • Compared with SecondPose, F2G-Pose additionally predicts complete camera-frame shape for dense supervision and uses MoE encoding for conditional capacity across categories.Source: §2 · p. 3
  • Instance-level methods use CAD models or reference views and generally target instance-specific reconstruction or alignment, unlike category-level pose-size estimation under a fixed taxonomy.F2G-Pose uses category-level canonical supervision during training and requires only a segmented RGB-D observation at inference for the evaluated taxonomy.Source: §2 · p. 3

3 Method

F2G-Pose predicts pose, metric size, and camera-frame completed shape from an RGB crop and segmented partial point cloud in one feed-forward pass. It lifts dense visual features onto 3D points, encodes local and object-level structure, and trains its direct pose-size and shape branches with geometric and pose supervision.

3 Method

  • Given an RGB crop and partial point cloud, F2G-Pose directly predicts 6D pose, metric size, and camera-frame completed shape in one feed-forward pass.It does not require instance-specific CAD models, reference images, or explicit category labels.Source: §3 · p. 3

3.1 Geometry-Aware Foundation Feature Lifting

  • Dense RADIOv2.5 features are projected onto observed 3D points, then fused with point geometry by DGCNN to form local input tokens.The resulting embedding has 128 tokens with dimension 512.Source: §3.1 · pp. 3–4

3.2 MoE-Based Object Encoding

  • A MoE Transformer integrates local token evidence into a global object representation for metric prediction, using top-2 routing among eight experts.A soft load-balancing penalty encourages balanced gate probabilities during training.Source: §3.2 · p. 4

3.3 Direct Pose-Size and Shape Decoding

  • Parallel decoder branches directly predict pose and size, and complete the shape in the camera frame; joint training lets dense geometry supervise the shared representation.The shape branch refines coarse geometry by fusing sampled observations and predicting local offsets.Source: §3.3 · pp. 4–5

3.4 Training Objectives

  • Rotation and size supervision uses Smooth L1 losses, with rotation compared against the closest symmetry-equivalent ground-truth target.The targets cover no symmetry, 90-degree, 180-degree, and arbitrary-angle rotational symmetry.Source: §3.4 · p. 5
  • Chamfer losses supervise coarse and dense shape outputs against the same complete ground-truth point cloud, while an auxiliary loss encourages balanced MoE expert use.The full objective combines pose-size prediction, camera-frame shape completion, and MoE load balancing, with λ = 0.01.Source: §3.4 · p. 5

4 Experiments

Experiments evaluate pose-size accuracy, latency, camera-frame shape reconstruction, ablations, real-world transfer, and robot manipulation. F2G-Pose improves benchmark accuracy while running at 28 FPS for pose-only inference.

4.1 Comparison with State-of-the-Art Methods

  • On SOPE, F2G-Pose improves GenPose++ from 31.9 to 43.7 AUC at IoU50 and reduces mean rotation error from 19.9° to 15.7°; on ROPE it improves AUC across all IoU thresholds.Pose-only inference runs at 28 FPS, compared with 4 FPS for GenPose++; full-model inference runs at 24 FPS.Source: §4.1 · pp. 6–7
  • 8° median versus 39° mean rotation error and 26.9% of candidates above 45° expose GenPose++'s heavy-tailed ROPE candidate set.With K = 50, an oracle finds a candidate within 5° for 79.3% of instances; reducing candidates improves responsiveness but lowers final accuracy.Source: §4.1 · p. 7, Table 2 · p. 7, Figure 3 · p. 8
  • F2G-Pose achieves substantially lower camera-frame Chamfer-L1 error than geometry-only completion baselines on SOPE and HouseCat6D.Depth points anchor visible metric surfaces, while lifted RGB features provide cues for completing occluded regions.Source: §4.1 · pp. 7–8, Table 3 · p. 8

4.2 Ablation Studies

  • Removing RGB features causes the largest ROPE ablation drop, while DGCNN tokens, Transformer context, and shape supervision each contribute to the full model's performance.DINOv2 or CLIP causes only modest IoU50 degradation relative to RADIO, suggesting gains stem primarily from architecture and supervision.Source: §4.2 · pp. 8–9, Table 5 · p. 8
  • Top-2 MoE offers a favorable accuracy-latency trade-off; load balancing improves routing entropy, load concentration, and pose accuracy over unbalanced top-2 routing.Top-1 is faster but less accurate, while top-4 yields only marginal gains at higher latency cost.Source: Table 6 · p. 8, §4.2 · p. 9
  • F2G-Pose degrades moderately on transparent and reflective objects, with reflective objects more difficult and IoU75 particularly sensitive to imperfect depth.Depth holes, specular artifacts, and missing thin structures can perturb estimated object extent.Source: §4.2 · p. 9, Table 7 · p. 9

4.3 Real-World Evaluation and Deployment

  • On HANDAL, F2G-Pose outperforms reference-free Any6D and GenPose++; on ROPE, it surpasses FoundationPose and Any6D without an instance reference.Low IoU75 values indicate that high-precision box alignment remains challenging under cross-dataset sensing and annotation differences.Source: §4.3 · p. 9, Table 8 · p. 9
  • ARX trials yield 74/80 successful picks and 68/80 placements; full GenPose++ sampling and clustering exceeds the 50 ms control budget.Failures include corrupted depth removing thin or asymmetric parts, severe occlusion of functional geometry, and symmetric low-texture objects admitting 90°/180° alternatives.Source: §4.3 · p. 10, Table 9 · p. 10

5 Conclusion

F2G-Pose jointly estimates category-level pose, metric size, and camera-frame shape from one RGB-D observation, balancing accuracy, latency, and manipulation success across evaluations. Its segmented-input requirement and sensitivity to depth corruption, occlusion, and ambiguous rotations bound its current scope.

5 Conclusion

  • Experiments on SOPE, ROPE, HouseCat6D, HANDAL, and ARX manipulation show a favorable balance of accuracy, inference latency, and manipulation success.The shape branch supplies dense geometric supervision and approximate planning geometry.Source: §5 · p. 10
  • F2G-Pose assumes a segmented RGB-D crop; mask failures and severe depth corruption can propagate to pose, size, and shape.High-precision metric alignment remains weak under cross-dataset shift and may require downstream refinement for tight geometric tolerances.Source: §5 · p. 10
  • The paper identifies possible benefits for assistive manipulation and human–robot collaboration, while urging human supervision for deployment in surveillance or autonomous systems.The broader-impact discussion also identifies those applications as potential risks without adequate safety oversight.Source: §5 · p. 10

A.1 Model Details

  • The method predicts pose, translation, metric size, and a 2048-point completed shape from fused RGB-D point features using a DGCNN encoder and Transformer blocks with top-2 Mixture-of-Experts layers.The shape branch combines predicted and observed points before selecting and refining candidates.Source: §A.1 · p. 15

A.2 Training Details

The task-specific network is trained on the SOPE training split, while runtime comparisons use a shared inference protocol and hardware setup.

A.2 Training Details

  • F2G-Pose is trained only on the SOPE training split for 100 epochs with batch size 512 using AdamW.Training uses four NVIDIA H200 GPUs, PyTorch 2.4, CUDA 12.4, automatic mixed precision, DistributedDataParallel, and synchronized BatchNorm.Source: §A.2 · p. 15
  • Runtime comparisons use the same masks, RGB-D crops, depth preprocessing, batch size 1, and GPU-synchronized timing on one RTX 4090.The pose-only setting disables the shape head after the shared encoder; the full setting retains it.Source: §A.2 · p. 15

A.3 Controlled Robustness Evaluation

The controlled ROPE perturbation study finds greater sensitivity to depth noise than photometric changes, while GenPose++ candidate quality varies with object symmetry. These analyses are framed as stress tests and candidate diagnostics, not a separate dataset contribution.

A.3 Controlled Robustness Evaluation

  • The ROPE robustness evaluation applies RGB photometric changes and depth perturbations as stress tests of the SOPE-trained model.Figure 5 illustrates controlled RGB and depth perturbations.Source: §A.3 · p. 15, Figure 5 · p. 16
  • Photometric performance changes mildly under light and moderate perturbations, with a larger but bounded drop under heavy perturbations.The tested RGB changes include exposure, white balance, and synthetic illumination; Table 10 reports the results.Source: §A.3 · p. 15, Table 10 · p. 16
  • Depth noise affects F2G-Pose more strongly than photometric shifts, consistent with sensitivity of metric alignment to corrupted or missing depth.Table 11 summarizes results for the tested heteroscedastic noise levels.Source: §A.3 · p. 15, Table 11 · p. 16

A.4 Additional GenPose++ Candidate Diagnostics

  • GenPose++ candidate quality is uneven across object structures: non-rotationally symmetric objects have fewer near-perfect candidates and larger average rotation error, making post-hoc selection harder.The oracle best-of-50 result indicates that accurate hypotheses are often present in the candidate set; Table 12 breaks results down by symmetry.Source: Table 12 · p. 16, §A.4 · p. 16

A.5 Failure Mode Analysis

Qualitative failures arise when depth corruption or occlusion removes important geometry, and when symmetry or weak texture leaves orientation ambiguous.

A.5 Failure Mode Analysis

  • ROPE failure cases include corrupted depth on thin or reflective structures, heavy occlusion, and ambiguous rotations for symmetric or weakly textured objects.These cases are consistent with the paper's stated limitations.Source: §A.5 · p. 17
  • Reflective-region examples show failures associated with corrupted or incomplete depth.Figure 6 presents these cases.Source: Figure 6 · p. 17
  • Symmetric, geometrically regular, or small objects can have ambiguous orientation cues.Figure 7 illustrates these cases.Source: Figure 7 · p. 17

A.6 Supplementary Ablation Results

The supplementary analyses examine ablation choices, qualitative examples, shape completion, mask sensitivity, and expert routing. They identify design trade-offs and input constraints relevant to the shared pose, size, and shape predictions.

A.6 Supplementary Ablation Results

  • Top-2 MoE is the adopted routing trade-off; top-4 adds only marginal gain at higher cost, while top-1 is faster but less accurate.RADIO performs best among the tested backbones, and point selection and coarse-to-fine refinement also affect pose-size accuracy through the shared representation.Source: §A.6 · p. 17, Table 13 · p. 18

A.7 Additional Qualitative Results

  • Additional ROPE examples compare input RGB, ground truth, GenPose++ predictions, and F2G-Pose predictions; HouseCat6D and HANDAL examples extend the qualitative comparisons.The HouseCat6D examples include reflective and transparent objects with depth corruption and camera-frame shape predictions; HANDAL compares ground-truth boxes with Any6D and F2G-Pose.Source: §A.7 · p. 18, Figure 8 · p. 19, Figure 9 · p. 20

B.1 Matched RGB-D Shape Completion

  • 36.7 →34.1 for F2G-Pose and 37.2 →35.5 for AdaPoinTr show that pose-size supervision improves canonical completion for both architectures.Under full supervision, F2G-Pose achieves 41.5 versus 38.5 IoU50 AUC; using canonical rather than camera-frame completion reduces F2G-Pose pose IoU50 AUC from 43.7 to 41.5. Canonical and camera-frame CD values are not directly comparable.Source: §B.1 · p. 21, Table 14 · p. 21

B.2 Segmentation-Mask Sensitivity

  • At matched mask IoU, erosion generally degrades pose metrics and shape CD more than dilation, while SAM masks yield results close to the GT-mask baseline.Erosion removes foreground evidence and dilation includes background; these sensitivity tests do not remove the requirement for segmented RGB-D input.Source: §B.2 · pp. 21–22, Table 15 · p. 21, Table 16 · p. 21

B.3 MoE Expert Preferences

  • Across 7,466 samples, routing is associated with category (Cramér’s V = 0.306, p < 0.001), with overlapping preferences for broad geometry-related groups rather than a rigid one-expert-per-category partition.Material and visibility associations are weaker, and no significant independent association with symmetry was found.Source: §B.3 · p. 22

B.4 HANDAL Selection and Per-Category Results

HANDAL evaluation uses seven categories with direct semantic matches to the SOPE taxonomy. Results cover 92 instances and indicate coarse alignment under cross-dataset transfer, while precise joint estimation remains challenging.

B.4 HANDAL Selection and Per-Category Results

  • Seven of HANDAL’s 17 categories are evaluated because they have direct semantic correspondences to SOPE; selection is based on taxonomy, not model performance.The seven categories map to six SOPE categories, with both wrench types mapped to spanner; categories without clear SOPE counterparts are excluded because no corresponding canonical frame is defined.Source: §B.4 · p. 22
  • 35.0, 11.7, and 0.3 are the macro-average AUCs at IoU25, IoU50, and IoU75, respectively, over 92 HANDAL instances.The results indicate coarse 3D alignment under cross-dataset transfer, while precise joint estimation of pose, size, and extent remains challenging; evaluation is seen-category, unseen-instance transfer.Source: §B.4 · p. 22, Table 17 · p. 22

B.5 Reference-Free and Single-Reference Evaluation Protocols

The protocols distinguish reference-free evaluation from evaluation with one additional view of the same object. The reference-based methods use that view under their stated inference procedures.

B.5 Reference-Free and Single-Reference Evaluation Protocols

  • Any6D’s zero-reference protocol supplies no additional reference view and uses its reference-free image-to-3D pathway to reconstruct a coarse shape before estimating pose.Source: §B.5 · p. 22
  • Any6D’s one-reference protocol randomly samples one additional view of the same object from its ROPE multi-view sequence and uses standard reference-query inference.Source: §B.5 · p. 22
  • FoundationPose is evaluated through its model-free pathway with one randomly sampled additional real view, from which an object mesh is reconstructed for pose estimation.Source: §B.5 · p. 22

Related papers

Loading 2610.31623v1…