Source-linked AI summary
BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects
Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas Muller, Alex Evans, Dieter Fox, Jan Kautz, Stan Birchfield
TL;DR
The paper addresses monocular RGBD 6-DoF tracking and 3D reconstruction of unknown objects. It jointly performs online pose optimization and neural reconstruction, achieving state-of-the-art results while handling challenging conditions, though combined difficulties, depth sensing failures, and non-rigid objects remain limitations.
Problem
Monocular RGBD 6-DoF pose tracking and 3D reconstruction of unknown objects are fundamental computer-vision problems supporting applications including augmented reality and robotic manipulation.
Method
The method jointly runs online pose graph optimization and Neural Object Field reconstruction, using a memory pool and hybrid SDF representation while requiring only initial-frame segmentation for rigid objects.
Results
Experiments on three public benchmarks demonstrate state-of-the-art performance, including new benchmark records for ADD-S and reconstruction chamfer distance and comparable ADD performance.
Takeaways & Limitations
Joint tracking and reconstruction improve the method’s supported robustness across severe motion, occlusion, limited texture, specular highlights, and difficult benchmark settings.
Takeaways & Limitations
The method can fail when severe occlusion, segmentation error, limited texture, and weak geometric cues occur together; it also requires depth sensing and assumes rigid objects.
Abstract
from arXiv · showhide
We present a near real-time method for 6-DoF tracking of an unknown object from a monocular RGBD video sequence, while simultaneously performing neural 3D reconstruction of the object. Our method works for arbitrary rigid objects, even when visual texture is largely absent. The object is assumed to be segmented in the first frame only. No additional information is required, and no assumption is made about the interaction agent. Key to our method is a Neural Object Field that is learned concurrently with a pose graph optimization process in order to robustly accumulate information into a consistent 3D representation capturing both geometry and appearance. A dynamic pool of posed memory frames is automatically maintained to facilitate communication between these threads. Our approach handles challenging sequences with large pose changes, partial and full occlusion, untextured surfaces, and specular highlights. We show results on HO3D, YCBInEOAT, and BEHAVE datasets, demonstrating that our method significantly outperforms existing approaches. Project page: https://bundlesdf.github.io
1. Introduction
The paper jointly addresses unknown-object 6-DoF tracking and 3D reconstruction from monocular RGBD video, targeting difficult conditions that separate approaches often cannot handle. Its online design combines pose optimization, neural reconstruction, and memory to improve robustness.
- 6-DoF pose tracking and 3D reconstruction of unknown objects are presented as closely related computer-vision problems with applications in augmented reality and robotics.
- Prior reconstruction methods commonly assume known camera poses or ground-truth masks, while tracking methods often require textured 3D object models.
- The method jointly solves tracking and reconstruction for rigid objects using only a first-frame 2D mask, while allowing free motion and severe occlusion.
- An online pose graph, concurrent Neural Object Field, and memory pool communicate to accumulate a consistent representation of object geometry and appearance.
- The approach is designed for occlusion, specularity, limited texture and geometric cues, and abrupt object motion, with robustness highlighted in Fig. 1.
- Experiments on three public benchmarks demonstrate state-of-the-art performance against leading methods.
2. Related Work
Related work treats unknown-object tracking and object reconstruction as challenging problems with substantial input assumptions. Dynamic interaction differs from static-scene settings by exposing additional object faces for reconstruction.
- 6-DoF Object Pose Estimation and Tracking: State-of-the-art 6-DoF tracking methods often require instance- or category-level CAD models, while newer methods may require posed reference views of the test object.
- 6-DoF Object Pose Estimation and Tracking: Object-level SLAM typically studies static scenes, whereas interaction introduces occlusions and missing cues but can expose different object faces for more complete reconstruction.
- Object Reconstruction: Learning-based image reconstruction and neural scene representations can produce high-quality 3D models, but their assumptions limit direct application to the paper’s setting.
3. Approach
The approach causally tracks an unknown rigid object’s 6-DoF pose from monocular RGBD video while reconstructing a textured 3D model, using only the first-frame object mask. It combines coarse correspondence-based initialization, a selective memory pool, online pose-graph optimization, and concurrent Neural Object Field learning.
- Overview: The system uses first-frame-only segmentation to track the object pose and reconstruct its textured 3D model causally from monocular RGBD video.It requires no object-specific CAD model or category-level prior, and does not require a particular amount of visual texture.
- Coarse Pose Initialization: Feature correspondences between consecutive segmented RGB frames, combined with depth and RANSAC filtering, provide each current frame’s coarse pose estimate.The pose hypothesis with the most inliers is selected before later refinement.
- Memory Pool: A keyframe memory pool retains informative historical observations, beginning with the first frame as the canonical coordinate system and adding frames that enrich viewpoint diversity.Frames are compared using rotational geodesic distance while ignoring in-plane rotation, allowing sparser memory allocation without reducing comparable multi-view consistency information.
- Online Pose Graph Optimization: For each new frame, no more than K selected memory frames form a pose graph whose joint optimization refines the current and participating historical poses.When the pool contains at most K frames, all frames are used; otherwise selection seeks maximum multi-view consistency information.
- Neural Object Field: Neural Object Field optimization runs in a separate thread over memory frames, updating their poses and feeding the learned field back into later pose optimization.Frames updated by the Neural Object Field are thereafter excluded from online pose-graph pose modifications, while other frames can continue receiving online updates.
- Online Pose Graph Optimization: The pose-graph objective combines pairwise feature and geometric losses with an optional unary neural-shape loss, using robust Huber penalties for correspondence distances.The Neural Object Field unary term is activated only after its initial training converges, with its weights frozen during this optimization step.
4. Experiments
Experiments evaluate pose tracking and 3D reconstruction across three real-world datasets and challenging interaction settings. The method outperforms comparison approaches, maintaining robust tracking and achieving strong reconstruction, especially under occlusion and limited visual cues.
- Datasets: Evaluation covers HO3D, YCBInEOAT, and BEHAVE, using RGBD videos of human or robot interaction with rigid objects.The datasets vary in interaction type, camera viewpoint, occlusion, motion, object properties, and sensing distance.
- Evaluation protocol: Pose tracking is measured with ADD and ADD-S AUC percentages, while reconstruction uses chamfer distance in the first-frame canonical coordinate frame.The comparison includes DROID-SLAM, NICE-SLAM, KinectFusion, BundleTrack, and SDF-2-SDF with RGBD video and the first-frame object mask as input.
- HO3D results: Our method outperforms comparison methods by a large margin on both 6-DoF pose tracking and 3D reconstruction on HO3D.Object-centric baselines have fewer texture or geometric cues available, leading to poor performance in this setting.
- HO3D results: Our method maintains low tracking error throughout HO3D videos, while BundleTrack has similarly low translation error but struggles with rotation estimation.Figure 5 reports rotation error using geodesic distance and separately reports translation error, averaging each timestamp across videos.
- Qualitative results: Our method tracks accurately and reconstructs higher-quality shapes despite severe occlusion, self-occlusion, limited texture, and strong lighting reflections.Qualitative comparisons also report cases where the predicted pose is more accurate than the multi-camera ground-truth annotation.
- Cross-dataset results: The method sets benchmark records on ADD-S and chamfer distance, achieves comparable ADD performance to the previous state-of-the-art, and outperforms prior methods on BEHAVE.BundleTrack achieves competitive pose tracking but unsatisfactory reconstruction, supporting the reported benefit of jointly designing tracking and reconstruction.
- Ablation study: Ablations show dramatically worse performance without memory, while replacing the Neural Object Field with handcrafted-loss optimization struggles on objects with little texture or geometric cues.The study also evaluates removing the Neural Object Field and hybrid SDF components.
5. Conclusion
The paper presents a monocular RGBD method that jointly tracks and reconstructs an unknown rigid object using only its initial segmentation. Online pose optimization and Neural Object Field learning support robust performance across challenging motion, occlusion, appearance, and texture conditions.
- Conclusion: The method requires only initial-frame object segmentation and uses parallel online graph pose optimization and Neural Object Field representation.It is designed for monocular RGBD video and rigid objects.
- Conclusion: Across several datasets, the method achieves state-of-the-art results while handling fast motion, partial and complete occlusion, low texture, and specular highlights.The reported scope concerns simultaneous 6-DoF object tracking and 3D reconstruction.
- Conclusion: Future work will use shape priors to reconstruct unseen object parts.
A. Implementation Details
Implementation details describe memory-based pose initialization and optimization together with neural-volume normalization and periodic network reinitialization. These choices target efficiency, refinement, and robustness to missing detections, occlusion, and noisy poses.
- Pose initialization: When the previous frame is unavailable, coarse pose initialization compares the current frame with memory frames and selects one having more than 10 feature correspondences.The selected memory frame becomes the new reference, after which the usual initialization steps continue.
- Pose optimization: Online pose graph optimization uses at most K = 10 memory frames, rejects correspondences beyond 1 cm or 20° normal-angle difference, and runs seven Gauss–Newton steps.
- Neural Object Field: The neural volume is normalized to [−1, 1] using a scale equal to 1.5 times the initial point-cloud dimension and coordinates centered on the first-frame point cloud.The geometry network uses a two-layer MLP with hidden dimension 64 and ReLU activations except at the last layer.
- Neural Object Field: Network weights are reinitialized for new training periods because reusing them can become trapped in local minima caused particularly by noisy poses.Reinitialization takes a similar number of steps to refining newly added memory-frame poses.
B. Computation Time
The method uses concurrent tracking and neural reconstruction threads, achieving near-real-time online tracking at approximately 10.2 Hz on standard desktop hardware.
- The online tracking thread processes frames at around 10.2 Hz.Segmentation, coarse matching, and pose graph optimization average 18 ms, 24 ms, and 56 ms, respectively.
- The neural object field thread runs in the background, requiring 6.7 seconds on average per training round.It exchanges data with the main thread after each training round.
- On the same hardware, DROID-SLAM runs at 6.1 Hz and BundleTrack at 11.2 Hz.
C. Metrics
Pose evaluation uses AUC percentages for ADD and ADD-S, while reconstruction evaluation uses symmetric Chamfer distance between reconstructed and ground-truth meshes.
- Pose metrics: Pose estimation is evaluated using the area under the curve percentage of ADD and ADD-S metrics.
- Pose metrics: ADD measures average pointwise Euclidean distance between transformed object-model points under estimated and ground-truth poses.
- Pose metrics: ADD-S uses nearest-neighbor correspondences between transformed model points, supporting symmetric pose evaluation.
- Pose metrics: The canonical coordinate frame for each video is defined by the ground-truth pose in the first image because the unknown object’s CAD model is inaccessible.
- Shape metric: Shape reconstruction is evaluated with symmetric Chamfer distance between the final reconstructed mesh and ground-truth mesh.Meshes are sampled at the same 5 mm resolution, and never-visible ground-truth faces are culled before evaluation.
D. Detailed Results
Across three datasets, the method generally achieves the strongest pose-tracking results, with AUC exceeding other methods for almost all datasets and best performance on most videos.
- Quantitative results: The method’s ADD-S and ADD AUC exceeds that of other methods for almost all datasets.Recall curves are reported for HO3D, YCBInEOAT, and BEHAVE.
- Quantitative results: The method performs best on almost all HO3D videos, more than half of YCBInEOAT videos, and a large majority of BEHAVE videos.
- Qualitative results: Qualitative results are presented across HO3D, YCBInEOAT, and BEHAVE, with supplemental video recommended for further examples.
- Evaluation setup: For BEHAVE, all methods are evaluated using a single monocular camera input rather than the dataset’s four-camera capture.
- Evaluation setup: The BEHAVE camera with the least sequence-wide occlusion is automatically selected using average object visibility.Visibility is computed offline by comparing ground-truth and rendered object masks.
E. Robustness Analysis
The method remains robust across weak visual cues, severe occlusion, specularity, abrupt motion, segmentation noise, and noisy depth while maintaining tracking and reconstruction quality.
- Dearth of Texture or Geometric Cues: The method handles texture-poor and geometrically ambiguous surfaces that challenge optical-flow, keypoint-matching, and photometric-loss methods.
- Occlusions: The method maintains robustness under self-occlusion and external occlusion from human interaction.
- Specularity: The method keeps tracking accurately on specular metallic or highly smooth surfaces where comparison methods become brittle.
- Abrupt Motion and Motion Blur: Under abrupt object motion and RGB motion blur, the method remains robust and even yields more accurate pose than ground-truth.
- Noisy Segmentation: The downstream modules maintain accurate tracking despite false-positive and false-negative segmentation masks.
- Noisy Depth: Simultaneous pose tracking and reconstruction produce a high-quality mesh despite noisy depth sensing and segmentation contamination.
F. Limitation and Failure Modes
The method remains robust across several difficult conditions, but combined challenges can cause tracking failure. Its scope is also limited by depth sensing and the rigid-object assumption.
- Failure modes: Severe occlusion, segmentation error, sparse texture, and weak geometric cues together can cause tracking failure, with symmetric geometry affecting the recovered pose after reappearance.The paper illustrates this combined-challenge failure in Fig. 18.
- Scope boundaries: The method requires depth input, limiting application to objects such as transparent objects where depth sensing fails.
- Scope boundaries: The method assumes the object is rigid and does not yet generalize simultaneously to rigid and non-rigid objects.The paper identifies joint rigid and non-rigid handling as future work.
- Observed robustness: Qualitative evaluations show robustness to little texture or geometric cues, severe or complete occlusion, fast pose changes, motion blur, and noisy segmentation.These conditions are illustrated on HO3D, YCBInEOAT, and BEHAVE examples.