PaperlayerView exact revision on arXiv

Source-linked AI summary

LEGAU: Learning Semantic Gaussian Priors for Scalable Category-level Pose Estimation

Hongli Xu, Zhaowei Lu, Junwen Huang, Jiaqi Hu, Peter KT Yu, Benjamin Busam, Federico Tombari, Slobodan ilic

arXiv:2609.35046v1cs.CV

TL;DR

Category-level pose estimation must infer object structure and pose from partial observations, while existing approaches often separate pose from shape understanding. LEGAU jointly predicts canonical correspondence, pose, size, and a category-conditioned Semantic Gaussian Field; experiments report improved performance in a single-model multi-category setting, with real-world transfer still challenging under strict pose thresholds.

  • Problem

    Category-level pose estimation methods often model pose and shape separately, despite the need to interpret partial observations using shared structural priors across object categories.

    Source: §1 · p. 2

  • Method

    LEGAU jointly predicts NOCS correspondence, object pose and size, and a canonical Semantic Gaussian Field that supplies category-conditioned structural cues for pose-shape reasoning.

    Source: §1 · pp. 2–3

  • Results

    Experiments on HouseCat6D, SOPE, and ROPE show improved performance in a single-model multi-category setting, while real-world transfer remains challenging under strict pose thresholds.

    Source: §6 · p. 13

  • Takeaways & Limitations

    Controlled ablations indicate that the Gaussian field and category-conditioned feature cues contribute to pose accuracy beyond geometry-only reconstruction.

    Source: §6 · p. 13

    The task assumes a cropped RGB image and its corresponding depth-derived partial point cloud as input.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Category-level 6D pose estimation from a single RGB-D observation is inherently under-constrained, since partial visible geometry must be interpreted together with a canonical object structure before a stable pose can be determined. We present LEGAU, a unified framework that jointly predicts NOCS correspondence, object pose and size, and a canonical Semantic Gaussian Field. Rather than treating reconstruction as a detached auxiliary task, LEGAU uses the Gaussian field as a category-conditioned structural prior that participates in multimodal feature fusion and provides global guidance for local pose reasoning. Conditioned on a categorical text embedding, LEGAU processes RGB-D observations through a transformer-based fusion module that integrates visual, geometric, and category-level cues, decoding the NOCS map, pose and size information and the Gaussian-based object representation. Extensive experiments on synthetic and real-world benchmarks show that this coupled pose-shape formulation achieves strong performance in a single-model multi-category setting, with up to 22\% on SOPE and competitive transfer to real-world data. These results highlight the benefit of jointly learning canonical correspondence, object shape, and pose alignment within a unified representation.

1 Introduction

Category-level pose methods must infer structure from partial observations, but pose and shape are often modeled separately, limiting their handling of occlusion and shape variation. LEGAU frames pose estimation as coupled pose-shape inference, using a category-conditioned Semantic Gaussian Field to jointly support correspondence, pose, and shape reasoning.

1 Introduction

  • Prior methods often separate pose estimation from shape understanding, leaving pose sensitive to occlusion, incomplete geometry, and substantial intra-class shape and appearance variation.Some reconstruction-and-alignment approaches also depend heavily on visible regions, while certain global-prior methods require per-category shape-reconstruction training.Source: §1 · p. 2
  • The introduction frames the central question as whether shape, semantics, and pose reasoning can be unified in one representation that directly supports pose alignment.The motivation is that reconstruction, semantic understanding, and pose reasoning are inherently coupled through alignment with the observed input.Source: §1 · p. 2
  • LEGAU fuses visual-geometric local embeddings, category text cues, and latent field embeddings through a multimodal Transformer for joint canonical correspondence, shape, and pose learning.The field provides global structural guidance for local pose reasoning.Source: §1 · p. 3
  • LEGAU formulates category-level pose estimation as coupled inference over canonical correspondence, object shape, and SE(3) pose, rather than independent pose regression and reconstruction.The Semantic Gaussian Field models canonical geometry and category-conditioned feature cues as a structured prior for pose-shape coupling under partial observations.Source: §1 · p. 3

2 Related Work

Related work spans category-level pose estimation, object reconstruction for pose, and joint shape-pose learning. LEGAU is positioned as a category-conditioned framework that couples these tasks through a shared representation and avoids per-category reconstruction networks.

Category-Level Object Pose Estimation

  • Earlier category-level methods use NOCS for generalization to unseen instances, while later methods add semantic or geometric priors but often keep shape and pose as separate branches.Other designs use explicit canonical models or do not fully couple reconstruction with pose reasoning.Source: §2 · p. 3

Object Reconstruction for Pose Estimation

  • Recent 3D-GS pose methods, including GS-Pose, 6D-GS, and 6DOPE-GS, lack category-level priors and reconstruct visible regions rather than complete object shapes.Generated shapes used by Giga-Pose and Any6D may hallucinate geometry or appearance.Source: §2 · p. 4

Joint Learning of Object Shape and Pose

  • Joint shape-pose frameworks often refine pose and shape alternately, and some 3D-GS methods lack global priors for complete-shape generation under limited viewpoints.The related category-agnostic approach instead predicts pose, size, and dense shape without category labels; LEGAU focuses on category-conditioned coupling.Source: §2 · p. 4
  • LEGAU unifies NOCS prediction, 6D pose regression, and semantic Gaussian field reconstruction in one multimodal Transformer and shared model, avoiding per-category reconstruction networks.The related-work summary contrasts pose methods lacking holistic 3D reasoning with reconstruction methods that overlook spatial and semantic alignment to partial observations.Source: §2 · p. 4

3 Method

LEGAU takes a partial RGB-D observation and category prompt, then uses interacting local, text, and field representations to predict pose-related outputs and a canonical Gaussian representation. Its Semantic Gaussian Field supplies category-conditioned structural cues that participate in pose reasoning.

3.1 Overview

  • Given an RGB-D input and categorical text prompt, LEGAU fuses local visual-geometric, text, and global field embeddings through a Transformer.Local embeddings drive NOCS and pose decoders, while the global embedding guides Gaussian-based shape decoding under multi-level supervision.Source: §3.1 · pp. 4–5, Figure 2 · p. 5

3.2 Task Formulation.

  • The task is to jointly estimate object 6D pose, 3D size, normalized object coordinates, and a dense feature Gaussian representation from a cropped RGB image and depth-derived partial point cloud.The Gaussian representation is specified in canonical space.Source: §3.2 · p. 5

3.3 Preliminary: Feature 3D Gaussians

  • LEGAU introduces a learnable triplane Semantic Gaussian Field that captures canonical geometry, semantics, and part structure before Gaussian primitives are decoded.The field interacts with local visual and geometric features to guide NOCS prediction and pose estimation, with the stated aim of resolving ambiguity from occlusion or partial observations.Source: §3.3 · pp. 5–6

3.4 Multimodal Representation

LEGAU builds a shared latent representation from local RGB-D features, a category text token, and a learnable triplane field embedding. These complementary cues provide local geometry and appearance alongside global semantic-shape context for pose reasoning.

3.4 Multimodal Representation

  • LEGAU aligns sampled RGB and point-cloud features into local tokens, adds a category-label text token, and uses a learnable triplane field embedding for holistic geometry and semantic context.The local tokens support NOCS prediction, while the field embedding provides global structural guidance and regularizes pose reasoning for consistent alignment across unseen categories.Source: §3.4 · p. 6

3.5 Unified Transformer Backbone

LEGAU uses alternating attention to fuse triplane field and local embeddings with category-conditioned semantic priors, producing representations for Gaussian-field reconstruction and pose estimation.

3.5 Unified Transformer Backbone

  • The multimodal sequence combines three orthogonal triplane field embeddings with spatially grounded local visual-geometric embeddings.Together, these token groups are processed jointly by the transformer.Source: §3.5 · p. 6
  • Global, local, and category-conditioned attention respectively exchange information across token groups, refine spatial coherence, and inject semantic priors into all representations.Stacked layers support interaction between local and global context while preserving geometry-aware features for downstream pose estimation and reconstruction.Source: §3.5 · p. 7
  • The backbone outputs refined local embeddings and a global field embedding for Gaussian-field reconstruction and 6D pose estimation.The local tokens retain SE(3)-aware spatial detail, while triplane embeddings consolidate geometric structure and semantics.Source: §3.5 · p. 7

3.6 Gaussians Decoder

The Gaussians Decoder samples latent features from a triplane field at canonical grid centers, decodes Gaussian attributes, and renders view-dependent outputs.

3.6 Gaussians Decoder

  • The decoder uniformly samples canonical-volume grid centers, projects each onto three orthogonal triplane planes, and aggregates interpolated latent features.The plane features are obtained by bilinear interpolation at each projected location.Source: §3.6 · p. 7
  • A lightweight MLP decodes each sampled latent feature into Gaussian center, anisotropic scale, opacity, orientation, color, and learned feature attributes.These attributes let the triplane field encode latent 3D geometry in a compact form.Source: §3.6 · p. 7
  • Differentiable Gaussian splatting renders the predicted Gaussians from a camera pose into view-dependent RGB, depth, and feature maps.Rendered feature maps accumulate per-Gaussian embeddings along ray compositing, following RGB and opacity blending.Source: §3.6 · p. 7

3.7 NOCS and Pose Decoders

LEGAU decodes canonical NOCS coordinates from local embeddings, then combines those embeddings with predicted coordinates and depth points to estimate pose and size.

3.7 NOCS and Pose Decoders

  • The NOCS decoder regresses normalized object coordinates for each input point from local embeddings.This canonicalization encourages embeddings to encode consistent geometric structure across instances.Source: §3.7 · p. 8
  • Pose features concatenate MLP-encoded local embeddings, predicted NOCS coordinates, and depth points.This fusion constructs a pose-aware representation from learned features, canonical correspondence, and observed geometry.Source: §3.7 · p. 8
  • Independent heads regress translation and anisotropic size, while rotation is predicted using a continuous 6D representation.The pose outputs comprise rotation, translation, and 3D size.Source: §3.7 · p. 8

3.8 Training Objectives

LEGAU jointly supervises NOCS correspondence, pose alignment, and Gaussian-field reconstruction with a multi-task objective.

3.8 Training Objectives

  • NOCS coordinates use Smooth-L1 supervision, while rotation, translation, and size use L1 pose losses.The pose loss sums the three component losses.Source: §3.8 · p. 8
  • Gaussian-field reconstruction uses RGB and feature consistency, with RGB comparing rendered and target views and feature loss based on cosine similarity.The RGB term combines an L1 difference and an SSIM term.Source: §3.8 · p. 8
  • The training objective combines NOCS, pose, RGB reconstruction, and feature-consistency losses to jointly supervise correspondence, alignment, and field reconstruction.The stated total loss weights these terms with separate coefficients.Source: §3.8 · pp. 8–9
  • The multi-task formulation is described as regularizing local SE(3)-aware correspondence and global Gaussian-field semantics, supporting geometry-consistent pose estimates under occlusion or unseen category shifts.The stated scope includes occlusion and category shifts.Source: §3.8 · p. 9

4 Implementation Details

LEGAU uses a multimodal transformer and a category-conditioned Semantic Gaussian Field to jointly learn canonical correspondence, pose, and shape.

4 Implementation Details

  • The backbone has 12 alternating-attention layers and hidden dimension 256, fusing frozen DINOv2-Small RGB features with PointNet point-cloud features.The PointNet has 128 hidden units.Source: §4 · p. 9
  • The Semantic Gaussian Field uses a triplane latent structure with three 32×32×32 feature planes and 131,072 uniformly sampled canonical Gaussian centers.Training renders 518×518 RGB, depth, and feature images using an adapted Gaussian-splatting kernel.Source: §4 · p. 9
  • For each object, 42 multiview images are prerendered and four views are sampled per iteration to supervise the Gaussian field.The model uses 1,024 depth points per object and is trained end-to-end with AdamW for 200,000 iterations at learning rate 2×10^-4.Source: §4 · p. 9
  • The loss weights balance canonical correspondence, pose regression, and Gaussian-field reconstruction objectives.The weights are (λnocs, λpose, λa, λssim, λd, λf) = (2.0, 0.3, 5.0, 1.0, 0.02, 0.3); training uses cosine decay and batch size 64.Source: §4 · p. 9

5 Experiments

LEGAU is evaluated on synthetic and real-world category-level benchmarks, with experiments assessing cross-dataset performance, semantic priors, and shape reconstruction.

5.1 Datasets and Metrics

  • HouseCat6D covers 149 real-world instances across 10 categories, SOPE is synthetic with 149 categories, and ROPE contains real RGB-D captures from those same categories.LEGAU is trained on SOPE and evaluated on ROPE without finetuning; metrics include AUC and VUS on Omni6DPose and accuracy and IoU on HouseCat6D.Source: §5.1 · pp. 9–10

5.2 Cross-Dataset Evaluation

  • Figure 3 compares LEGAU with AGPose and GenPose++ on examples involving symmetry, occlusion, transparency, real-world novel objects, and specular surfaces.Source: Figure 3 · p. 10
  • LEGAU achieves strong performance on HouseCat6D and SOPE, while on ROPE it remains competitive in IoU metrics but has a gap under stricter pose thresholds.The SOPE-trained model is evaluated on ROPE without finetuning; Table 1 compares the three benchmarks.Source: §5.2 · pp. 10–11, Table 1 · p. 11

5.3 Effect of Learning Semantic Priors

  • The Semantic Gaussian Field shows category-level structural patterns, including smooth axial gradients for elongated objects and embeddings aligned with symmetry axes.Per-Gaussian semantic embeddings are rendered as RGB colors using PCA in Figure 4.Source: §5.3 · p. 11, Figure 4 · p. 12
  • Removing the Semantic Gaussian Field reduces HouseCat6D pose accuracy from 57.4 to 55.1 (-4.4%), while removing CLIP conditioning lowers SOPE VUS@10°5cm from 44.2 to 40.7 (-7.9%).The ablations isolate Gaussian reconstruction and CLIP-based category conditioning.Source: §5.3 · p. 12, Table 2 · p. 12

5.4 Shape Reconstruction Results.

  • On SOPE, LEGAU achieves the lowest Chamfer-L1 reconstruction error among the compared point-based methods.Figure 5 compares reconstructions with a depth-based baseline and image-to-3D models under heavy occlusion and transparency; LEGAU maintains geometric fidelity across viewpoints.Source: §5.4 · pp. 12–13, Table 3 · p. 12, Figure 5 · p. 13

6 Conclusion

LEGAU couples canonical correspondence, pose alignment, and Gaussian-based shape reconstruction through a category-conditioned structural prior; experiments show benefits in multi-category pose estimation, with real-world transfer still challenging under strict thresholds.

6 Conclusion

  • LEGAU treats category-level pose estimation as coupled inference over canonical correspondence, pose alignment, and Gaussian-based shape reconstruction.Its Semantic Gaussian Field acts as a category-conditioned structural prior that encourages pose estimates to remain consistent with object-level shape under partial observations.Source: §6 · p. 13
  • Experiments on HouseCat6D, SOPE, and ROPE show that pose-shape coupling improves performance in a single-model multi-category setting.Real-world transfer remains challenging under strict pose thresholds.Source: §6 · p. 13
  • Controlled ablations indicate that the Gaussian field and category-conditioned feature cues contribute to pose accuracy beyond geometry-only reconstruction.The paper identifies stronger open-vocabulary conditioning, broader real-world adaptation, and more interactive robotic perception as future directions.Source: §6 · p. 13

Related papers

Loading 2609.35046v1…