Source-linked AI summary
6-PACK: Category-level 6D Pose Tracker with Anchor-Based Keypoints
Chen Wang, Roberto Martín-Martín, Danfei Xu, Jun Lv, Cewu Lu, Li Fei-Fei, Silvio Savarese, Yuke Zhu
TL;DR
Category-level 6D tracking must handle novel instances without relying on known object models. 6-PACK learns compact 3D keypoints from RGB-D data and estimates interframe motion through their matching, achieving strong benchmark results and real-time robot deployment.
Problem
Category-level 6D tracking seeks to follow novel instances of known categories without requiring a known 3D object model.
Method
6-PACK uses anchors to generate a small ordered set of 3D keypoints, learned without manual supervision, and aligns consecutive sets to estimate pose changes.
Results
6-PACK substantially outperforms existing baselines on NOCS-REAL275 and runs at 10Hz in a robot deployment.
Takeaways & Limitations
The tracker supports real-time pose tracking and robot interaction, including manipulation tasks on a Toyota HSR platform.
Abstract
from arXiv · showhide
We present 6-PACK, a deep learning approach to category-level 6D object pose tracking on RGB-D data. Our method tracks in real-time novel object instances of known object categories such as bowls, laptops, and mugs. 6-PACK learns to compactly represent an object by a handful of 3D keypoints, based on which the interframe motion of an object instance can be estimated through keypoint matching. These keypoints are learned end-to-end without manual supervision in order to be most effective for tracking. Our experiments show that our method substantially outperforms existing methods on the NOCS category-level 6D pose estimation benchmark and supports a physical robot to perform simple vision-based closed-loop manipulation tasks. Our code and video are available at https://sites.google.com/view/6packtracking.
I. INTRODUCTION
6-PACK addresses category-level 6D pose tracking without requiring a known 3D model, using compact learned keypoints and temporal pose updates. It improves benchmark performance and supports real-time robot interaction.
- Category-level 6D tracking targets novel instances within known categories when acquiring perfect 3D geometry is difficult.The task extends pose tracking beyond instance-specific models.
- 6-PACK tracks a small set of RGB-D keypoints and accumulates relative pose changes over time.The method avoids defining and estimating absolute pose directly at every frame.
- The learned keypoints are discovered without manual supervision and provide a compact representation for efficiently estimating interframe motion.Keypoint correspondences between adjacent frames support pose-change estimation.
- 6-PACK substantially outperforms all baselines on NOCS-REAL275, including registration-based and tracking-by-detection methods.The comparison includes traditional registration, state-of-the-art tracking-by-detection, and ablative baselines.
- The tracker runs at 10Hz on a GTX1070 GPU and is deployed on a Toyota HSR for real-time manipulation tasks.The deployment demonstrates robot use of the tracking output.
II. RELATED WORK
Prior category-level pose methods use templates, dense correspondences, or manually selected landmarks, each introducing model, noise, supervision, or scalability constraints. 6-PACK instead learns a compact tracking-oriented keypoint set end-to-end without direct keypoint supervision.
- Template-matching pose methods degrade under clutter or variable lighting because feature-matching errors corrupt correspondences.
- Methods that match inputs to renderings or silhouettes require known object models and therefore remain instance-specific.
- Supervised category-level keypoints require substantial labeled data, while manual landmarks or bounding-box corners may not be optimal for tracking.
- NOCS projects every object pixel into a canonical 3D space, but dense correspondences are susceptible to occlusion noise.
- 6-PACK generates a small robust keypoint set using end-to-end pose-tracking supervision rather than direct keypoint annotations.
III. PROBLEM DEFINITION
Category-level 6D pose tracking estimates each object’s pose change between consecutive frames and recursively composes those changes from a given initial pose. The method represents motion with rotation and translation in SE(3).
- Given an initial pose and object category, tracking estimates the pose change ∆p_t between consecutive timesteps.
- Each pose change contains a rotation change ∆R_t in SO(3) and a translation change ∆t_t in R3.
- The absolute pose is recovered recursively by composing the latest estimated change with the previous pose.
- The setup assumes the initial pose is given, while the method is reported to remain robust to errors in that initialization.
- Matching keypoints from two consecutive frames enables least-squares recovery of the interframe pose change under a rigid-body assumption.
IV. MODEL
6-PACK uses anchors around a predicted pose to constrain 3D keypoint generation, then estimates motion from ordered keypoints in consecutive frames. The resulting pose update recenters the anchors for continued tracking.
- Overall workflow: The model applies anchor-based keypoint generation to the previous and current frames, then computes their interframe pose change.
- Anchor-based attention: Anchors are generated on a normalized volume cropped around the predicted object pose from RGB-D input.
- Anchor-based attention: Distance-weighted pooling summarizes RGB-D point features at anchor locations, and attention selects an anchor near the object centroid.
- Anchor-based attention: The selected anchor feature guides generation of an ordered keypoint set instead of searching the entire unconstrained 3D space.
- Overall workflow: Least-squares optimization estimates motion from current and previous keypoints, and the extrapolated pose centers the next anchor distribution.
- Initialization: An iterative initialization procedure refines a potentially inaccurate starting pose by repeatedly aligning it with generated-keypoint centroids.
A. Anchor-based Attention Mechanism
6-PACK uses a 3D grid of anchors to localize the object coarsely before generating keypoints, reducing the search from unconstrained 3D space to a region around the object centroid.
- 3D keypoint generation is difficult because objects may appear anywhere within the RGB-D field of view.
- The method adapts 2D detection’s anchor strategy by selecting an anchor near the object center before generating a more accurate keypoint proposal.
- Each anchor pools fused color and geometric features from nearby RGB-D points using distance-weighted averaging.
- An attention network assigns confidence scores to anchors and selects the one closest to the object centroid during evaluation.
B. Unsupervised 3D Keypoints Generation
6-PACK generates ordered 3D keypoints from anchor features and trains them without manual keypoint annotations. Its losses enforce temporal consistency, pose-estimation utility, separation, surface proximity, centroid alignment, and symmetry-aware behavior.
- B. Unsupervised 3D Keypoints Generation: The keypoint network converts the selected anchor feature into an ordered K × 3 set of 3D keypoints for tracking.
- B. Unsupervised 3D Keypoints Generation: Keypoints are learned without manual annotations by optimizing multi-view consistency across consecutive frames.
- B. Unsupervised 3D Keypoints Generation: The differentiable pose loss ensures generated keypoints support recovery of the ground-truth inter-frame pose change rather than merely matching image locations.
- B. Unsupervised 3D Keypoints Generation: Separation and silhouette losses discourage degenerate keypoint configurations and encourage keypoints to remain near the object surface.
- B. Unsupervised 3D Keypoints Generation: A centroid loss aligns the generated keypoint centroid with the object centroid, and the overall objective combines six weighted loss terms.
- B. Unsupervised 3D Keypoints Generation: For symmetric categories, a rotation-invariant coordinate transformation replaces absolute rotation around the symmetry axis with relative inter-keypoint angles.
V. EXPERIMENTS
The experiments evaluate 6-PACK against multiple baselines on NOCS-REAL275, test robustness and keypoint design choices, and assess real-time robot deployment. 6-PACK achieves strong tracking performance, maintains stability over time, and supports closed-loop manipulation.
- Benchmark Results: 6-PACK outperforms NOCS by more than 15% in the 5°5 cm metric and 12% in the IoU25 metric.The comparison is reported on the NOCS-REAL275 evaluation.
- Stability: 6-PACK remains more than 10% higher than NOCS throughout the tracking process and stops decreasing approximately 100 frames after initialization.The stability analysis measures mean accuracy after excluding increasingly many initial frames, which are easier because the initial pose is given.
- Ablation: The 8-keypoint model achieves 62.4% in the 5°5cm metric, surpassing the 4-keypoint and 16-keypoint variants at 55.2% and 48.6%.The result identifies 8 keypoints as the best trade-off between information compression and redundancy for the laptop category.
- Robot Deployment: On a Toyota HSR, 6-PACK tracks poses at 10 Hz and supports manipulation tasks including pouring or tossing with bowl and bottle objects.The deployment uses an NVIDIA GTX1070 GPU and an RGB-D sensor, and also tests tracking on laptop and can instances.
VI. CONCLUSION
6-PACK tracks category-level object pose by generating consistent keypoints across instances and estimating inter-frame pose changes without manual supervision. It achieves state-of-the-art benchmark performance and supports real-time robot interaction.
- 6-PACK uses an anchor-based keypoint generation network to detect the same keypoints across different instances of one category.The network is trained unsupervised so it can select keypoints that are effective for tracking.
- The detected keypoints estimate inter-frame pose changes for category-level 6D object tracking.
- 6-PACK achieves state-of-the-art performance on a challenging category-based 6D object pose tracking benchmark.The evaluation compares 6-PACK with 3D geometry methods and deep learning models.
- The tracker enables real-time tracking and robot interaction on an HSR robot platform.