PaperlayerView exact revision on arXiv

Source-linked AI summary

BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models

Bowen Wen, Kostas Bekris

arXiv:2108.00516v1cs.CVcs.AIcs.GRcs.RO

TL;DR

BundleTrack addresses 6D pose tracking for novel objects when instance- or category-level 3D models are unavailable. It combines learned segmentation and feature extraction with memory-augmented pose-graph optimization, achieving strong benchmark results, including 87.4% under NOCS “5°5cm” versus 33.3% previously best.

  • Problem

    Existing 6D pose tracking often assumes instance or category-level 3D models, limiting generalization to novel objects and requiring model databases that are costly to build.

  • Method

    BundleTrack combines video segmentation, learned keypoint detection and matching, and memory-augmented pose-graph optimization using historical keyframes instead of a fused global model.

  • Results

    BundleTrack significantly improves over existing state-of-the-art methods across two public benchmarks, reaching 87.4% under NOCS “5°5cm” versus the previous best of 33.3%.

  • Takeaways & Limitations

    The framework provides long-term, low-drift tracking of novel objects without 3D models and achieves comparable performance to methods using instance CAD models.

Abstract

from arXiv · show

Tracking the 6D pose of objects in video sequences is important for robot manipulation. Most prior efforts, however, often assume that the target object's CAD model, at least at a category-level, is available for offline training or during online template matching. This work proposes BundleTrack, a general framework for 6D pose tracking of novel objects, which does not depend upon 3D models, either at the instance or category-level. It leverages the complementary attributes of recent advances in deep learning for segmentation and robust feature extraction, as well as memory-augmented pose graph optimization for spatiotemporal consistency. This enables long-term, low-drift tracking under various challenging scenarios, including significant occlusions and object motions. Comprehensive experiments given two public benchmarks demonstrate that the proposed approach significantly outperforms state-of-art, category-level 6D tracking or dynamic SLAM methods. When compared against state-of-art methods that rely on an object instance CAD model, comparable performance is achieved, despite the proposed method's reduced information requirements. An efficient implementation in CUDA provides a real-time performance of 10Hz for the entire framework. Code is available at: https://github.com/wenbowen123/BundleTrack

I. INTRODUCTION

BundleTrack targets accurate, robust 6D tracking of novel objects without instance- or category-level 3D models. It combines learned segmentation and feature matching with memory-augmented pose-graph optimization, achieving strong benchmark performance and real-time execution.

  • Motivation: Instance-level 3D models complicate generalization to novel instances, while category-level approaches remain limited by training-category coverage and costly database construction.Existing databases such as ShapeNet and ModelNet40 contain 55 and 40 categories, respectively, and may require scanning, mesh refinement, or CAD design.
  • Approach: BundleTrack combines video segmentation, learned keypoint detection and matching, and memory-augmented pose-graph optimization for model-free 6D tracking.Historical observations remain as representative keyframes in a memory pool rather than being fused into a global model.
  • Results: Experiments on two public benchmarks report significant qualitative and quantitative improvement over methods using instance-level or category-level models and SLAM-like methods.The framework is evaluated under challenging tracking conditions described in the introduction.
  • Approach: Memory-augmented optimization supports low-drift tracking by enabling multi-hop data association and retaining raw keyframes instead of aggregating observations into a global model.The memory pool supplements correspondences between consecutive frames and allows previously estimated poses to contribute to later optimization.
  • Efficiency: A CUDA implementation enables online execution of computationally heavy feature matching and pose-graph optimization.The implementation is presented as an efficiency contribution for the complete tracking framework.
  • Results: 87.4% accuracy under the “5°5cm” metric on NOCS improves the previous best of 33.3%, including comparison with category-level-model methods.The reported improvement is from 33.3% to 87.4% on the NOCS Dataset.

II. RELATED WORK

Related work includes model-based 6D tracking, dynamic object-aware SLAM, and hand-held object scanning. These approaches can require object models, accumulate reconstruction errors, or depend on forward kinematics in ways that limit generalization.

  • 6D Object Pose Tracking: Instance-model 6D tracking uses handcrafted filtering, optimization, or machine learning, but requiring instance-level models complicates generalization to novel instances.The model may be needed for offline training or model-frame registration during tracking.
  • Dynamic Object-aware SLAM: Dynamic object-aware SLAM reconstructs objects on the fly while tracking them without pre-existing object models.These methods combine frame-model ICP with color, probabilistic association, or 3D level-set likelihood maximization.
  • Dynamic Object-aware SLAM: Tracking-via-reconstruction can transfer pose-estimation errors into the reconstructed model, adversely affecting subsequent tracking.BundleTrack instead maintains observed frames as pose-graph nodes so previously erroneous estimates can be corrected and long-term drift reduced.
  • 3D Hand-held Object Scanning: A robot-manipulation scanning framework simultaneously reconstructs and tracks objects, but also requires forward kinematics and therefore does not generalize when objects are not held by the manipulator.Its reconstruction-and-tracking formulation has issues similar to dynamic SLAM methods.

III. PROBLEM FORMULATION

The problem is to continuously track a rigid object’s relative 6D pose from RGB-D data without an instance model or category-level model database. BundleTrack uses an initial object mask and optionally an initial camera-frame pose to define the tracking task.

  • Problem Definition: The objective is to track a rigid object’s relative pose transformation T0→τ ∈ SE(3) over time from the start of tracking.The input is a sequence of RGB-D frames and an initial binary mask M0 indicating the object region.
  • Problem Definition: The formulation assumes neither a corresponding instance 3D model nor a category-level 3D model database for training.This assumption defines the model-free tracking setting.
  • Initialization: The initial mask M0 may come from semantic or non-semantic segmentation, point-cloud clustering, or plane fitting and removal.The formulation permits multiple initialization mechanisms rather than requiring a specific segmentation source.
  • Framework Overview: BundleTrack propagates an object mask, extracts keypoints and descriptors, matches consecutive frames for a coarse relative transform, optimizes selected keyframes, and stores novel views in memory.The framework’s stages are shown from left to right in Fig. 2.
  • Pose Representation: An optional initial camera-frame pose TC0 enables recovery of absolute camera-frame pose; otherwise, the tracked output is defined relative to the initial frame.The relative transformation is the quantity directly computed by tracking.

IV. APPROACH

BundleTrack propagates object masks through video and combines learned feature matching with pose-graph optimization for causal 6D tracking. Its segmentation module requires only an initial binary mask and can be replaced when simpler object-region estimates are available.

  • Tracking pipeline: BundleTrack crops object regions from consecutive frames, extracts keypoints and descriptors, matches features, and prunes outliers before pose optimization.The current mask and previous mask define the regions sent to the keypoint network.
  • Tracking pipeline: At most K memory keyframes participate in pose-graph optimization, balancing computational efficiency against tracking accuracy.Feature and geometric correspondences are computed in parallel on the GPU.
  • Object segmentation: Video segmentation propagates the previous object mask to the current frame, avoiding independently processed masks and temporal inconsistencies.The method uses dense long-term similarity dependencies between current and past feature embeddings.
  • Object segmentation: The segmentation output is a binary object mask distinguishing the target from the background, initialized by M0.M0 need not come from semantic or instance segmentation.
  • Object segmentation: The segmentation network is replaceable when an alternative method can compute the object region, such as forward-kinematics-based ROI extraction followed by point-cloud filtering.The subsequent BundleTrack stages do not depend on transductive-VOS specifically.

B. Keypoint Detection, Matching and Local Registration

BundleTrack estimates an initial relative pose from generalizable learned features rather than category-specific keypoints, then selects mutually informative historical keyframes for refinement.

  • Keypoint detection and local registration: LF-Net provides generalizable keypoints and descriptors trained on general 2D images rather than instance- or category-specific 3D models.The choice balances feature performance with inference speed.
  • Keypoint detection and local registration: RANSAC-based feature matching and outlier pruning produce a preliminary pose from correspondences between consecutive frames.The preliminary pose is initialized from the previous pose and the estimated relative transformation.
  • Keyframe selection: Keyframe selection chooses up to K historical frames with the largest mutual viewing overlap when the memory pool exceeds K.The selection problem is formulated as a minimum H-subgraph of an edge-weighted graph.
  • Keyframe selection: The edge weight between keyframes is the geodesic distance between their rotations, so minimizing rotation differences maximizes mutual viewing overlap.An iterative greedy procedure starts from the initial frame and adds keyframes until K are selected.

D. Online Pose Graph Optimization

Online pose-graph optimization jointly refines the current pose and selected historical keyframes using feature and geometric constraints. The optimized current pose is output causally, while corrected historical poses improve future initialization and reduce drift.

  • Pose graph construction: The graph contains the current pose and selected historical keyframes, with feature and geometric energies defined between node pairs.The graph minimizes total energy to enforce spatiotemporal consistency.
  • Graph energies: Feature correspondences are reused when available or computed between node pairs, while multi-pair matching is parallelized on the GPU.The feature energy uses unprojected 3D points and a Huber M-estimator.
  • Graph energies: Geometric correspondences use dense pixel re-projection, filtering outliers by point distance and surface-normal angle.Perspective projection and depth-based unprojection connect image pixels to 3D points.
  • Optimization: The optimization balances feature and geometric energies with λ1 and λ2, both set to 1 in the experiments.The weights are reported as insensitive to their choice.
  • Optimization: Gauss-Newton updates optimize 6D Lie-algebra pose vectors for the current frame and selected keyframes while holding the initial frame fixed.Each pose has three translation and three rotation parameters.
  • Optimization: A preconditioned conjugate-gradient solver exploits Jacobian and weight-matrix sparsity during each Gauss-Newton step.Incremental updates accumulate in the tangent space.
  • Online output: The current optimized pose becomes the causal output, whereas corrected historical poses remain in memory to improve initialization and reduce long-term drift.Past output poses are not retroactively changed.

E. Augmenting the Keyframe Memory Pool

The keyframe memory pool is expanded only when the current view is sufficiently novel relative to all stored keyframes, enriching multi-view diversity without adding redundant frames.

  • Keyframe insertion: The initial frame is always retained, while a later frame is added only if its rotation distance from every stored keyframe exceeds α.The experiments use α = arccos(10°).
  • Keyframe insertion: This insertion rule encourages keyframes from novel views, enriching the memory pool’s multi-view diversity.The criterion compares the current pose against every existing keyframe.

V. EXPERIMENTS

BundleTrack is evaluated on two public benchmarks spanning unseen objects, static-object tracking with a moving camera, and robot-manipulation scenarios. It achieves comparable or superior performance without instance or category-level 3D models.

  • BundleTrack is evaluated on the NOCS and YCBInEOAT public benchmarks across diverse objects and tracking scenarios.NOCS uses static tabletop objects and a moving camera, whereas YCBInEOAT evaluates dynamically moving objects during robot manipulation.
  • BundleTrack uses no CAD models or 3D-object-database training data in the experiments.
  • BundleTrack achieves comparable or superior performance relative to alternatives despite requiring no instance or category-level 3D models.

B. Results on the NOCS Dataset

On NOCS, BundleTrack is compared with category-prior, model-free, and SLAM-based methods under a noisy-initialization and missing-frame protocol. The evaluation reports threshold accuracy, overlap, rotation error, and translation error, with qualitative comparisons shown in Fig. 3.

  • Comparisons include category-level-prior methods, model-free methods, and dynamic SLAM approaches, with qualitative results presented in Fig. 3.
  • The protocol perturbs initialization by up to 4cm, drops 450 of 3,200 frames, and forbids re-initialization.
  • The evaluation uses 5°5cm and IoU25, where higher is better, plus Rerr and Terr, where lower is better.
  • YCBInEOAT results use ADD and ADD-S AUC metrics for comparisons involving instance-level, category-level, and no-model methods.

D. Analysis

The analysis studies pose-graph components, initialization sensitivity, runtime, drift, generalization, and failure cases. BundleTrack runs at 10Hz, tracks without retraining, and can recover from some occlusion-induced drift, while segmentation errors remain a limitation.

  • Ablation Study: The ablation study evaluates online global pose-graph optimization and each energy term.
  • Computation Time: 10Hz is the average runtime of the entire framework, including video segmentation.Related methods run at 4Hz, 11Hz, and 17Hz on the same machine for 6-PACK, TEASER++*, and MaskFusion respectively.
  • Generalization: BundleTrack keeps fixed neural-network weights and hyperparameters across evaluations without retraining or fine-tuning.It requires neither instance nor category-level 3D models for training or registration on novel instances.
  • Failure Cases: Memory-augmented pose-graph optimization helps recover tracking after severe self-occlusion when later frames provide better visibility.
  • Failure Cases: Noisy segmentation can omit object regions, prevent feature extraction, and produce a slight translation offset.

VI. CONCLUSION

BundleTrack tracks novel-object 6D pose without instance or category-level 3D models and supports long-term accurate tracking in challenging scenarios. Its performance is comparable to state-of-the-art CAD-dependent methods.

  • BundleTrack is a general framework for tracking novel-object 6D pose without instance or category-level 3D models.
  • The framework performs long-term accurate tracking under various challenging scenarios.
  • BundleTrack achieves comparable performance to state-of-the-art methods that depend on the target object’s CAD model.
Loading 2108.00516v1…