PaperlayerView exact revision on arXiv

Source-linked AI summary

NovaSplat: Model-Free 6D Object Pose Estimation via 3D Gaussian Splatting Onboarding and Refinement

Georgios Floros, Evangelos Sartinas, Orestis Vangelis, Maria Pateraki

arXiv:2610.13309v1cs.CVcs.RO

TL;DR

Model-free 6D pose estimation must build object representations from onboarding video and has historically trailed CAD-based methods. NovaSplat uses a 3DGS representation for templates and pose refinement, surpassing model-based methods on HOPEv2 and outperforming model-free competitors on IndustryShapes while narrowing the model-based gap.

  • Problem

    Model-free 6D pose estimation must reconstruct object representations from short videos and has historically lagged methods that use CAD models.

    Source: §1 · p. 1

  • Method

    NovaSplat trains a 3DGS representation from onboarding video for template-based estimation and differentiable pose refinement with combined appearance and geometric cues.

    Source: §1 · pp. 1–2, §3.5 · p. 4

  • Results

    2.9% on AP and 17.0% on APMSSD over the best model-based baseline on HOPEv2; NovaSplat also beats the best model-free competitor on IndustryShapes while narrowing the model-based gap.

    Source: §5 · p. 8

  • Takeaways & Limitations

    NovaSplat-Lite is 3.1× faster at a cost of 0.001 AP, and the results show model-free estimation can surpass model-based methods on HOPEv2 without CAD models.

    Source: §5 · p. 8

    Textureless, specular surfaces on IndustryShapes degrade reconstruction quality and feature matching, leaving a performance gap to model-based methods.

    Source: §5 · p. 8

Abstract

from arXiv · show

6D pose estimation of novel objects without CAD models remains an open challenge: model-free methods must build the object representation from scratch during onboarding and have long trailed their model-based counterparts. We present NovaSplat, a model-free pipeline that closes this gap. From a short smartphone video captured at onboarding, we train a 3D Gaussian Splatting (3DGS) model that serves both as a template source for detection and coarse pose estimation and as a differentiable backbone for pose refinement. Our refiner optimizes the object-to-camera pose with combined photometric and geometric losses under a continuous rotation parametrization free of gradient singularities. On a household-object benchmark, NovaSplat surpasses all model-based methods. On a challenging industrial benchmark with textureless and specular surfaces, it outperforms all model-free competitors while substantially narrowing the gap to model-based ones. We further introduce NovaSplat-Lite, an inference variant that is $3.1\times$ faster at negligible accuracy cost. Code will be made available at \href{https://github.com/POSE-Lab/novasplat}{https://github.com/POSE-Lab/novasplat}.

1. Introduction

NovaSplat addresses the difficulty of model-free 6D pose estimation by using an onboarding video to build a 3DGS representation for both template-based estimation and pose refinement. It targets the performance gap with model-based methods, which typically require unavailable CAD models.

1. Introduction

  • Model-based pose methods rely on CAD models that are rarely available outside controlled environments, motivating model-free estimation from a short onboarding video.6D pose estimation recovers an object's full position and orientation relative to a camera.Source: §1 · p. 1
  • NovaSplat surpasses all model-based methods on HOPEv2 and all model-free competitors on IndustryShapes, narrowing the gap to model-based methods there.Figure 1 compares APMSSD on HOPEv2 and ARMSSD on IndustryShapes.Source: Figure 1 · p. 1
  • Model-free methods must reconstruct geometry from noisier, less complete video, which degrades downstream stages and has historically left them behind model-based methods.The passage links reconstruction quality to template rendering and correspondence matching.Source: §1 · p. 1
  • NovaSplat trains a 3DGS object representation from video for high-quality templates and uses it as a differentiable backbone for gradient-based pose refinement.The refiner jointly minimizes appearance and geometric losses using a continuous rotation parameterization that avoids singularities.Source: §1 · pp. 1–2
  • The paper presents a complete model-free onboarding pipeline and a 3DGS-based refiner, reporting state-of-the-art results across household and industrial benchmarks.Its stated contribution is that model-free methods can match or exceed model-based performance without CAD models.Source: §1 · p. 2

2. Related Work

Related work distinguishes pose methods by their object knowledge and inference requirements, then situates NovaSplat among model-based, model-free, and neural-rendering approaches. NovaSplat builds on correspondence-based estimation while using 3DGS for templates and refinement without a CAD model.

2.1. Taxonomy of 6D Object Pose Estimation

  • Pose-estimation methods are categorized by prior object knowledge: instance-based methods use 3D models during training, while novel-object methods include model-based and model-free unseen-object approaches.Model-based approaches require a CAD model at inference; model-free approaches build a representation from an onboarding video.Source: §2.1 · p. 2

2.2. Model-Based Novel Object Methods

  • Model-based novel-object methods match features or correspondences over CAD-rendered templates; NovaSplat builds on FreeZeV2 and replaces ICP refinement with a differentiable 3DGS-based refiner.The cited approaches use geometric priors from rendered templates and point clouds, and all require a CAD model at inference.Source: §2.2 · p. 2

2.3. Model-Free Novel Object Methods

  • Model-free approaches include SfM-based localization, template retrieval, and few-shot methods, which can struggle with low texture, viewpoint changes, detection quality, or restrictive assumptions.Examples of assumptions include symmetry priors and scale constraints.Source: §2.3 · p. 2

2.4. Neural Rendering for Object Pose Estimation

  • Neural-rendering pose methods use differentiable object representations; prior 3DGS methods often evaluate on simpler benchmarks, while GFreeDet2 is closest but uses visual features alone.NovaSplat combines rendered templates and mesh-derived geometry for coarse estimation, then refines pose with photometric and geometric losses.Source: §2.4 · pp. 2–3

3. Method

NovaSplat builds object assets from a short multi-view video, then detects objects, estimates a coarse pose, and refines it using a frozen 3DGS representation. The pipeline combines visual and geometric evidence from onboarding through inference.

3.2. Onboarding

  • Onboarding uses SfM, 3DGS training, and MVS to produce a renderable 3DGS representation and a mesh-derived template point cloud from a short multi-view video.The 3DGS-densified Gaussian centers provide a semi-dense prior for MVS, which fuses depth maps into a point cloud and mesh.Source: §3.1 · p. 3, §3.2 · p. 4

3.1. Overview

  • Inference detects and segments each object, estimates a coarse pose, then refines it; onboarding assets are reused throughout these stages.The query is an RGB-D image containing instances of onboarded objects.Source: §3.1 · p. 3

3.3. Detection and Segmentation

  • Detection replaces SAM/FastSAM with YOLO-E and DINOv2 with DINOv3, using rendered templates rather than CAD-rendered templates or masked onboarding frames.The passage describes these as extensions to the CNOS propose-then-match workflow.Source: §3.3 · p. 4

3.4. Coarse Pose Estimation

  • Coarse pose estimation fuses visual and geometric descriptors into per-point matches, then uses RANSAC to produce the initial pose.Symmetry-aware augmentation is moved to this stage, where all template views provide more viewpoint variation.Source: §3.4 · p. 4

3.5. 3DGS-Based Pose Refinement

  • The refiner freezes the learned Gaussian parameters and optimizes only object-to-camera pose using appearance and geometric cues with a continuous, nonsingular rotation representation.The photometric loss uses SSIM and MS-SSIM; the geometric loss is a bidirectional Chamfer distance between template and query point clouds.Source: §3.5 · p. 4

4. Experiments

NovaSplat is evaluated on household and industrial benchmarks, with experiments covering inference cost, benchmark performance, and the effects of refinement and onboarding choices. It outperforms model-free competitors on both benchmarks, while its results against model-based methods vary by dataset and metric.

4.1. Datasets

  • HOPEv2 covers cluttered household scenes, while IndustryShapes adds industrial objects with textureless surfaces, specular reflections, thin geometries, and small scale.The evaluation uses AP metrics for HOPEv2 and AR metrics for IndustryShapes, averaging metrics for each dataset's overall accuracy.Source: §4.1 · p. 5, §4.2 · p. 5

4.4. Main Results

  • On HOPEv2, NovaSplat exceeds GFreeDet2 by 17.0% on AP (0.529 vs. 0.452) and 39.5% on APMSSD (0.509 vs. 0.365).It also outperforms all model-based methods on AP and APMSSD, including Co-op (MUSE) by 2.9% on AP and 17.0% on APMSSD.Source: Table 1 · p. 5, §4.4 · pp. 6–7

4.3. Implementation Details

  • Onboarding takes 7.8 minutes; NovaSplat-Lite cuts inference from 56.1 s to 18.0 s per image, a 3.1× speed-up, at a 0.001 AP cost on HOPEv2.Lite keeps onboarding and the learned representation unchanged.Source: §4.3 · p. 6

4.4. Main Results

  • On IndustryShapes, NovaSplat surpasses all three model-free baselines on both splits, improving overall AR over FoundationPose by 44.0% on classic and 21.2% on extended.The largest margin is on ARMSSD (+64.5% on classic).Source: Table 2 · p. 7, §4.4 · p. 7
  • Co-op (MUSE) leads NovaSplat on HOPEv2 APMSPD (0.593 vs. 0.548), while FoundationPose retains higher IndustryShapes AR on classic and extended (0.67/0.69 vs. 0.36/0.40).On IndustryShapes ARMSSD, NovaSplat reduces the gap to the best model-based result from 0.43 to 0.23 on classic and from 0.37 to 0.27 on extended.Source: §4.4 · p. 7

4.5. Ablation Study

  • The full photometric-and-geometric refinement achieves 0.718 APMSSD and 0.736 AP, whereas geometry-only refinement lowers both metrics below the coarse-stage baseline.ICP also degrades performance: APMSSD falls from 0.703 to 0.384 and AP from 0.724 to 0.471.Source: Table 3 · p. 8, §4.5 · p. 8
  • Accuracy is essentially unaffected down to 25% of onboarding views, degrades moderately at 5%, and collapses at 1%.3DGS training accuracy improves up to 5k iterations and plateaus beyond that; 5k is recommended under the BOP onboarding budget.Source: §4.5 · p. 8

5. Conclusions

NovaSplat replaces the CAD-model requirement with onboarding from a short smartphone video and uses its 3DGS representation for templates and pose refinement. It reports strong benchmark results, while sensitivity to textureless, specular surfaces remains its primary limitation.

5. Conclusions

  • NovaSplat exceeded the best model-based HOPEv2 baseline by 2.9% on AP and 17.0% on APMSSD, and outperformed the best model-free IndustryShapes competitor.NovaSplat-Lite is 3.1× faster at a cost of 0.001 AP.Source: §5 · p. 8
  • SfM and MVS recover geometry from a short smartphone video to train a 3DGS representation for template rendering and differentiable pose refinement.Source: §5 · p. 8
  • Textureless, specular IndustryShapes objects degrade reconstruction and feature matching, leaving a gap to model-based methods.The authors identify appearance robustness as NovaSplat’s primary limitation.Source: §5 · p. 8

A. Full Implementation Details

NovaSplat’s full and Lite configurations share onboarding, the learned object representation, and detection, while differing in inference settings. Lite targets coarse pose estimation and refinement, reducing total inference time with negligible reported AP cost.

A. Full Implementation Details

  • The full and Lite configurations share onboarding, the learned object representation, and detection; Lite changes inference settings, including single-scale coarse estimation and a shorter refinement schedule.Timings use a workstation with an Intel i7-14700K and one RTX 4090; HOPEv2 timing covers 457 images averaging 21.6 instances, and reports both per-image and per-instance costs.Source: A. Full Implementation Details · p. 12, §B.1 · p. 12, §B.2 · p. 12

B.3. Where Coarse Pose Estimation Spends Its Time

  • GeDi descriptor extraction consumes 91.0% of Lite’s coarse-pose time and 49.3% of total per-image inference time.The remaining geometric verification steps together take 0.45 s per image, or 4.6% of the coarse-pose stage.Source: §B.3 · p. 13, Table 6 · p. 13

B.4. Optimizations

  • Lite reduces coarse pose estimation from 18.6 s to 9.8 s per image by keeping point clouds on the GPU and using one local-reference-frame scale.These changes leave the descriptor formulation unchanged.Source: §B.4 · p. 13
  • Lite reduces photometric refinement from 35.4 s to 7.0 s per image by lowering the iteration cap from 200 to 20 and early-stopping patience from 30 to 3.The authors report that the dominant correction occurs within the first few iterations from a well-initialized coarse pose.Source: §B.4 · p. 13
  • NovaSplat-Lite cuts inference from 56.1 s to 18.0 s per image, a 3.1× speed-up, at a cost of 0.001 AP on HOPEv2.The per-instance time falls from 2.60 s to 0.83 s; refinement is batchable across instances, making per-image timing a conservative upper bound for the stated static-scene setting.Source: §B.4 · p. 13

B.5. Memory Profile

Memory use is largely stable across the tested 3DGS training budgets, while a 5k-iteration budget substantially shortens onboarding. Accuracy plateaus at 5k iterations but falls at lower budgets.

B.5. Memory Profile

  • Onboarding typically uses 3.0–4.2 GB host and 1.6–3.1 GB device memory; its 18.0 GB host peak is a brief geometry-extraction and feature-export transient.Device memory rises in stages to 7.6 GB at onboarding’s end.Source: §B.5 · p. 14
  • Inference device memory reaches 6.0 GB early and stays flat, while host memory peaks at 8.0 GB after startup before declining to 2.1–3.0 GB in the run’s second half.Source: §B.5 · p. 14
  • Reducing 3DGS training from 14k to 5k iterations changes onboarding peaks by just 0.4 GB host and 0.1 GB device, while cutting the 28-object pass from 235 to 140 minutes.The figure shows the full HOPEv2 validation timeline; the memory envelope remains essentially unchanged.Source: §B.5 · p. 14, Figure 5 · p. 14

C. Onboarding Reconstruction Budget

  • At 5k iterations, HOPEv2 accuracy plateaus relative to 14k, while 1k and 500 iterations lose 4.1% and 11.0% AP, respectively.At 5k, AP is 0.741 vs. 0.736, APMSSD is 0.722 vs. 0.718, and APMSPD is 0.761 vs. 0.754; these sub-one-percent differences are treated as within run-to-run variation.Source: C. Onboarding Reconstruction Budget · p. 14, Table 7 · p. 14
  • The 5k setting reduces estimated onboarding to 258 s per object from 468.2 s, meeting the BOP protocol’s five-minute per-object budget.Source: C. Onboarding Reconstruction Budget · p. 14

D. Effect of Onboarding View Density

NovaSplat tolerates substantial onboarding-frame subsampling before accuracy declines, but extreme sparsity degrades rendered templates and pose accuracy. A separate protocol tests whether onboarding camera poses can be recovered from video alone, including for a textureless, specular object.

D. Effect of Onboarding View Density

  • At 25% of original views, AP and APMSSD match or slightly exceed baseline; at 5%, AP falls to 0.655, and at 1%, it collapses to 0.212.At 1%, rendered templates show severe artifacts and loss of detail; the authors connect the decline to insufficient multi-view coverage for SfM and 3DGS reconstruction.Source: D. Effect of Onboarding View Density · p. 15, Figure 6 · p. 15

E. Onboarding Poses from Structure-from-Motion

  • The pose-recovery test discards supplied camera poses and re-estimates the onboarding trajectory from RGB frames, including for the benchmark’s most textureless, specular object.The stated motivation is to test video-only recovery in a case where SfM is often expected to fail.Source: E. Onboarding Poses from… · p. 15

E.1. Protocol

  • The protocol uses COLMAP features and matches followed by GLOMAP, with every unmasked frame, fixed supplied intrinsics, and no loop detection or intrinsic refinement.It tests a textured HOPEv2 object and the most textureless, specular IndustryShapes object; both register fully into one model, though the latter’s cloud is two orders of magnitude thinner and 80% lies off the object.Source: §E.1 · p. 15, Table 9 · p. 16
  • Because the estimated trajectory has an arbitrary gauge, the protocol aligns camera centers to reference poses with Umeyama before reporting absolute errors; relative rotation error is alignment-invariant.Source: §E.1 · p. 15

E.2. Reconstruction

The onboarding reconstruction registers every selected frame, including on the textureless object, but yields far fewer sparse points there. Its estimated trajectory remains globally consistent, with higher absolute errors on the textureless sequence and an onboarding requirement for a feature-bearing scene.

E.2. Reconstruction

  • Every selected frame registers into one model with sub-pixel mean reprojection error, but the textureless reconstruction yields only 11 points per image versus 444 on HOPEv2.Only 1,117 of 5,802 points (19.3%) fall inside the object bounding box; 80% belong to the surrounding scene.Source: §E.2 · p. 16
  • The camera trajectory is constrained by the supporting surface, calibration target, and workspace, not by the object’s appearance; the estimated and reference trajectories overlap even for the textureless sequence.Ground-truth pose comparison covers 482 of 522 registered frames.Source: §E.2 · p. 16, §E.3 · p. 16, Figure 7 · p. 17

E.3. Pose Accuracy

  • On the textureless object, median pose error is 1.36° and 9.47 mm, versus 0.32° and 1.09 mm on the textured object; neither sequence has a failed frame or accumulating drift.The textureless sequence’s maximum position and rotation errors are 22.9 mm and 3.77°, respectively.Source: §E.3 · p. 16
  • The relative direction error is 13.95° on IndustryShapes versus 0.61° on HOPEv2, but the short consecutive-camera baseline makes direction poorly conditioned rather than indicating reconstruction failure.The IndustryShapes sequence is a single slow orbit across 522 frames, unlike the wider HOPEv2 trajectory across 178 frames.Source: §E.3 · p. 17

E.4. Discussion

  • The textureless sequence’s median position error is 9.47 mm, or 3.0% of the object diameter, below the tightest BOP MSSD threshold of 5%.The 95th-percentile position error is 4.7% of the diameter.Source: §E.4 · p. 17
  • Onboarding should occur in a feature-bearing environment, a scene-level requirement weaker than requiring texture on the object itself.The discussion states that the method does not depend on dataset-supplied onboarding poses and can be deployed on video captured in the wild.Source: §E.4 · p. 17

F. Per-Object Quantitative Results

Object 23 has NovaSplat’s lowest HOPEv2 validation performance, with near-symmetry making its canonical orientation difficult to distinguish and sometimes leading refinement to a symmetric counterpart.

F. Per-Object Quantitative Results

G. HOPEv2 Onboarding Data Inconsistencies

Some HOPEv2 objects differ physically between onboarding and query images, creating a mismatch between the reconstructed reference geometry and the observed object. This can affect refinement methods that rely on an explicit 3D reference.

G. HOPEv2 Onboarding Data Inconsistencies

  • HOPEv2 instances may have different physical configurations across onboarding and query images, including changes in the relative pose of articulated or loosely attached components.The example cap is sideways in onboarding but aligned above the printed label in validation.Source: G. HOPEv2 Onboarding Data… · p. 18, Figure 10 · p. 19
  • A changed component arrangement makes the onboarding-derived geometry inconsistent with the query image and can affect refinement against an explicit 3D reference.The rendered reference may contain local structures whose arrangement does not match the query observation.Source: G. HOPEv2 Onboarding Data… · p. 18
  • Methods relying primarily on global learned features or 2D image cues may be less affected by these local 3D discrepancies.Source: G. HOPEv2 Onboarding Data… · p. 18

H. Qualitative Comparisons

Qualitative comparisons show NovaSplat aligning predictions more closely with visible object extent and orientation than model-free baselines on HOPEv2, and producing competitive alignments on challenging industrial parts. False positives can persist when inherited from initial detections.

H. Qualitative Comparisons

  • On HOPEv2, NovaSplat’s predictions align more closely with visible object extent and orientation than competing model-free baselines, especially amid occlusion and clutter.The comparisons include multiple objects, partial occlusions, clutter, and visually similar packaging.Source: H. Qualitative Comparisons · p. 19, Figure 11 · p. 20
  • False positives may persist because NovaSplat attempts to align misdetections from the initial object detector rather than filtering them out.Explicitly filtering these inherited misdetections is identified as future work.Source: H. Qualitative Comparisons · p. 19
  • On IndustryShapes Classic and Extended, NovaSplat produces accurate pose alignments competitive with state-of-the-art model-based methods, including under extreme viewpoint variation.Source: H. Qualitative Comparisons · p. 19, Figure 12 · p. 21, Figure 13 · p. 22

Related papers

Loading 2610.13309v1…