PaperlayerView exact revision on arXiv

Source-linked AI summary

CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds

Yijia Weng, He Wang, Qiang Zhou, Yuzhe Qin, Yueqi Duan, Qingnan Fan, Baoquan Chen, Hao Su, Leonidas J. Guibas

arXiv:2104.03437v2cs.CV

TL;DR

Most category-level pose work targets single-frame estimation, while 6-PACK tracks only rigid-object 6D pose, leaving category-level 9DoF rigid and per-part articulated tracking as the target gap. CAPTRA canonicalizes point clouds, regresses residual rotations with RotationNet, and predicts dense normalized coordinates with CoordinateNet in an end-to-end tracking pipeline. On NOCS-REAL275 [29], CAPTRA improves mean accuracy of 5◦5cm over 6-PACK [28] by 40.03% absolute; on SAPIEN [34], it beats the single-frame baseline by around 20 points in eyeglasses.

  • Problem

    Most category-level pose work targets single-frame estimation, while 6-PACK tracks only rigid-object 6D pose, leaving category-level 9DoF rigid and per-part articulated tracking as the target gap.

    Source: §1 · p. 2

  • Method

    CAPTRA canonicalizes point clouds, regresses residual rotations with RotationNet, and predicts dense normalized coordinates with CoordinateNet in an end-to-end tracking pipeline.

    Source: §1 · p. 2

  • Results

    On NOCS-REAL275 [29], CAPTRA improves mean accuracy of 5◦5cm over 6-PACK [28] by 40.03% absolute; on SAPIEN [34], it beats the single-frame baseline by around 20 points in eyeglasses.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work demonstrates category-level pose tracking can cover both rigid objects and articulated parts within one framework.

    Source: §6 · p. 8

    Translation remains difficult because incomplete partial observations create translation ambiguity.

    Source: §4.2 · p. 4

Abstract

from arXiv · show

In this work, we tackle the problem of category-level online pose tracking of objects from point cloud sequences. For the first time, we propose a unified framework that can handle 9DoF pose tracking for novel rigid object instances as well as per-part pose tracking for articulated objects from known categories. Here the 9DoF pose, comprising 6D pose and 3D size, is equivalent to a 3D amodal bounding box representation with free 6D pose. Given the depth point cloud at the current frame and the estimated pose from the last frame, our novel end-to-end pipeline learns to accurately update the pose. Our pipeline is composed of three modules: 1) a pose canonicalization module that normalizes the pose of the input depth point cloud; 2) RotationNet, a module that directly regresses small interframe delta rotations; and 3) CoordinateNet, a module that predicts the normalized coordinates and segmentation, enabling analytical computation of the 3D size and translation. Leveraging the small pose regime in the pose-canonicalized point clouds, our method integrates the best of both worlds by combining dense coordinate prediction and direct rotation regression, thus yielding an end-to-end differentiable pipeline optimized for 9DoF pose accuracy (without using non-differentiable RANSAC). Our extensive experiments demonstrate that our method achieves new state-of-the-art performance on category-level rigid object pose (NOCS-REAL275) and articulated object pose benchmarks (SAPIEN, BMVC) at the fastest FPS ~12.

1. Introduction

CAPTRA tracks category-level 9DoF poses from live point-cloud streams for novel rigid objects and articulated-object parts. It combines pose canonicalization, residual rotation regression, and dense coordinate prediction to enable differentiable pose updates without RANSAC.

1. Introduction

  • CAPTRA tracks 9DoF poses continuously from a live point-cloud stream, given an initial pose, for rigid objects or each rigid part of articulated objects.The authors identify 6-PACK [28] as the closest related work, which tracks category-level 6D poses only for rigid objects.Source: §1 · p. 2
  • Coordinate-based methods can be accurate and robust but RANSAC is slow and non-differentiable, whereas direct pose regression is fast but error-prone, especially for arbitrary rotations.CAPTRA combines dense coordinate prediction with direct regression of small rotations in canonicalized point clouds.Source: §1 · p. 2
  • CAPTRA canonicalizes point clouds using previous-frame inverse poses, then RotationNet regresses small residual rotations while CoordinateNet predicts dense normalized coordinates for analytical size and translation computation.The combined pipeline is end-to-end differentiable and avoids non-differentiable RANSAC.Source: §1 · p. 2
  • On NOCS-REAL275, CAPTRA outperforms 6-PACK [28] by 40.03% absolute on mean accuracy of 5◦5cm and 10.52% absolute on mean IoU.Source: §1 · p. 2
  • On SAPIEN eyeglasses, CAPTRA beats the ANCSH [14]-based baseline by around 20 points on mean accuracy 5◦5cm; on BMVC laptops, it achieves the best performance without fine-tuning.Source: §1 · p. 2
  • CAPTRA achieves the fastest reported tracking speed among the compared methods, at ∼12 FPS, and experiments demonstrate robustness to pose errors.Source: §1 · p. 2

2. Related Works

Prior work establishes canonical coordinate spaces for category-level pose estimation, while existing tracking approaches either omit articulated objects and 3D size or rely on instance-specific and domain-specific assumptions.

Category-Level Object Pose Estimation

  • NOCS [29] defines a canonical frame for category-level rigid objects; NPCS [14] extends this representation to rigid parts of articulated objects.NOCS aligns category instances to a canonical orientation, zero-centers them, and uniformly scales their bounding boxes to diagonal length 1.Source: §2 · p. 2

Category-Level Object Pose Tracking

  • The only prior category-level tracker, 6-PACK [28], tracks rigid objects using keypoints but estimates only rotation and translation, omitting 3D size.It uses attention over anchors and unsupervised keypoint generation to compute interframe pose changes.Source: §2 · p. 3

Category-Level Object Pose Estimation

  • Methods improve NOCS using analysis-by-synthesis, shape generative models, and learnable deformation, but most use RANSAC and are not end-to-end differentiable.Differentiable alternatives include DSAC and DSAC++ [3].Source: §2 · p. 3

Instance-Level 6D Pose Tracking

  • Hand and human trackers use domain-specific knowledge, while instance-level trackers use CAD models, particle filtering, optimization, or learned delta-pose regression [15] [32].These instance-level methods track known object instances.Source: §2 · p. 3

3. Problem Definition and Notations

The section formulates online category-level 9DoF pose tracking for rigid and articulated objects, representing each part by its size, rotation, and translation. It assumes a known, constant number of rigid parts per category and estimates per-part poses from a live depth-point-cloud stream and pose initialization.

3. Problem Definition and Notations

  • The number of rigid parts M is known and constant within each category; M > 1 denotes an articulated category, while M = 1 denotes a rigid category [14].Source: §3 · p. 3
  • Each part pose is P^(j) = {d^(j), R^(j), T^(j)}, with 3D size d^(j), rotation R^(j) ∈ SO(3), and translation T^(j) ∈ R^3.The paper follows the category-level rigid-object and part-pose definition in [29, 14].Source: §3 · p. 3
  • Given a live depth-point-cloud stream, an object instance, and per-part pose initialization, the system estimates all part poses online for frames t > 0.At frame t+1, it uses the estimated poses from frame t together with the current depth point cloud.Source: §3 · p. 3

4. End-to-end Differentiable Pose Tracking

CAPTRA tracks rigid and articulated objects by canonicalizing pose inputs, regressing interframe rotations, and predicting part coordinates and segmentation to update pose. Its end-to-end differentiable pipeline avoids preprocessing and postprocessing.

4.1. Pose Canonicalization

  • Canonicalization transforms each part’s point cloud by the inverse estimated pose, constraining interframe updates around identity in category-level canonical frames and simplifying regression.Each part’s 9DoF pose is factorized into a 7DoF similarity transformation and a 3D aspect ratio.Source: §4.1 · pp. 3–4, Figure 2 · p. 4

4.2. Pose Tracking

  • RotationNet directly regresses small interframe rotations, while CoordinateNet predicts normalized coordinates and segmentation to resolve scale and translation more reliably than direct regression.Partial observations can make direct scale and translation regression ambiguous; normalized coordinates capture category-wise priors and support zero-centered translation estimation.Source: §4.2 · pp. 4–5

4.2.1 Rotation Estimation

  • Built on PointNet++, RotationNet regresses per-point, per-part rotations in a 6D continuous representation and averages predictions across each part using the Euclidean mean.At test time, CoordinateNet’s predicted part segmentation supplies the mask for rotation estimates.Source: §4.2.1 · p. 5

4.2.2 Scale and Translation Estimation

  • CoordinateNet uses a PointNet++ segmentation network to predict part labels and normalized coordinates; given RotationNet’s rotations, the method analytically computes per-part scale and translation with Umeyama.Only the first part’s pose-canonicalized point cloud is needed as input because canonicalized clouds for different parts share segmentation and normalized coordinates.Source: §4.2.2 · p. 5

4.3. Training Protocol

  • Training data are generated on the fly by perturbing ground-truth part poses with random Gaussian noise, and the total objective combines segmentation, coordinate, rotation, scale, translation, and corner losses.Direct pose losses are differentiable, including corner loss to improve 3D IoU predictions.Source: §4.3 · p. 5

5. Experiment

Experiments evaluate rigid tracking on NOCS-REAL275 and articulated tracking on SAPIEN and real-world trajectories, with benchmark comparisons, ablations, robustness tests, and speed measurements.

5.1. Datasets and Evaluation Metrics

  • The evaluation covers six rigid categories in NOCS-REAL275, four articulated categories in SAPIEN, and two real laptop sequences, using pose, box-overlap, and joint-state metrics as applicable.NOCS-REAL275 tests 3,200 frames of unseen instances; SAPIEN tests 19,000 frames of 19 unseen instances. Real laptop trajectories use AD accuracy with 10% of part diameter as the threshold.Source: §5.1 · pp. 5–6

5.2. Category-Level Rigid Object Pose Tracking

  • Our method achieves the best NOCS-REAL275 performance, outperforming 6-PACK under both initialization settings with less than a third of its rotation error and 105% higher 5°5cm accuracy.Depth-only translation performance is weaker because camera segmentation fails on noisy real depth images; excluding cameras, the method is best under all metrics. Unlike 6-PACK, it tracks full 9DoF pose and predicts bounding boxes.Source: §5.2 · p. 6, Table 1 · p. 6

5.3. Category-Level Articulated Pose Tracking

  • On held-out SAPIEN instances, our method outperforms the articulated-tracking baselines under all metrics, with especially precise rotation estimates despite ANCSH* using ground-truth labels and joint constraints.Per-part 6-PACK tracking was not run because the depth-only dataset does not provide the RGB-D input that 6-PACK requires.Source: §5.3 · p. 7, Table 2 · p. 7

5.4. Category-Level Articulated Pose Tracking on Real-World Data

  • On two real laptop trajectories, our SAPIEN-trained model outperforms the compared methods despite Sim2Real and category-level generalization gaps, and it tracks drawers during robot interaction.The real drawers trajectory was captured with Kinect2 as a robot pulled out the middle drawer; the paper also reports results on a captured real scissors trajectory.Source: §5.4 · p. 7, Table 3 · p. 7

5.5. Ablation Study

  • Ablations show that pose canonicalization improves CoordinateNet, RotationNet adds further gains over canonicalized CoordinateNet and CoordinateNet + DSAC++, and explicit pose losses improve performance.Canonicalized sRT regression performs worst; CoordinateNet variants use RANSAC-based pose fitting, and non-canonicalized CoordinateNet was not tested on articulated objects because of part ambiguity.Source: §5.5 · pp. 7–8, Table 4 · p. 7, Table 2 · p. 7

5.6. Tracking Robustness

  • Our method is significantly more robust to pose noise than 6-PACK when extra errors affect either initialization or every frame’s previous-pose estimate.The robustness test evaluates degradation in 5°5cm accuracy with one or two times the original pose errors added.Source: §5.6 · p. 8, Figure 4 · p. 8

5.7. Tracking speed

  • Our method has the highest tracking speed among the compared methods when tested on the same device.The paper attributes NOCS and ANCSH slowness to RANSAC and optimization, and reports that released 6-PACK code forwards its network 27 times per frame.Source: §5.7 · p. 8, Table 5 · p. 8

5.8. Discussions

  • Tracking scale reduces average scale error from 1.09% to 0.30% and raises articulated mIoU from 71.70% to 74.00%; rigid mIoU rises from 73.43% to 76.42%.The rigid comparison excludes cameras, where both schemes fail. In articulated tracking, inaccurate predicted joint constraints harmed performance, while ground-truth joint-axis constraints yielded little improvement; sensor noise causes failures on real depth images.Source: §5.8 · p. 8

6. Conclusion

CAPTRA introduces an end-to-end differentiable framework for category-level 9DoF pose tracking of rigid and articulated objects. It achieves state-of-the-art performance on both pose benchmark types and runs comparably fast at evaluation.

  • The paper presents category-level 9DoF pose tracking for both rigid and articulated objects as a first-time contribution.Source: §6 · p. 8
  • The end-to-end differentiable framework consists of pose canonicalization, RotationNet, and CoordinateNet.Source: §6 · p. 8
  • The algorithm achieves state-of-the-art performance on category-level rigid and articulated pose benchmarks and runs comparably fast for evaluation.Source: §6 · pp. 8–9

A. Coordinate-based Method Review · B. Proof of the pose canonicalization observation

The review explains how NOCS and NPCS encode canonical coordinates for category-level objects and how point correspondences yield object poses and sizes. A proof fragment concludes that a quantity is approximately zero as a natural result of temporal continuity.

A. Coordinate-based Method Review

B. Proof of the pose canonicalization observation

C. Principal Component Analysis of Residual Rotations … E.2. Training Details

These sections explain how CAPTRA handles residual rotations and symmetric-object ambiguities, computes per-part pose quantities, and configures its network inputs and training. They also describe rotation averaging and why small residual rotations are amenable to direct regression.

C. Principal Component Analysis of Residual Rotations

  • For small residual rotations, the first three principal components capture most output-space variance, supporting direct regression in the locally near-linear rotation space.The analysis samples 106 rotations across angle limits from 5° to 180° (unconstrained), using the 6D continuous representation.Source: Table 6 · p. 11, C. Principal Component Analysis of… · p. 11

D.1. Euclidean Mean of Rotations

  • To average multiple rotation predictions, CAPTRA converts their 6D representations to matrices, takes the Euclidean mean, and projects the result back to SO(3).The passage begins describing the use of a binary segmentation mask but does not provide the complete formula.Source: §D.1 · p. 11

D.2. Rotation Supervision for Symmetric Objects

  • For symmetric bowl, bottle, and can categories, rotation supervision regresses the unit symmetry-axis endpoint because only two rotation degrees of freedom are unique [29] [39].The paper uses this representation for those categories in NOCS-REAL275; articulated parts generally lack rotational ambiguity because their kinematic structures help disambiguate them.Source: §D.2 · p. 11

D.3. Coordinate Supervision for Symmetric Objects

  • For symmetric objects, coordinate supervision combines an L2 loss on the pairwise-distance matrix with a symmetric coordinate loss.Coordinates can rotate together around the symmetry axis without changing the object; pairwise distances remain invariant.Source: §D.3 · pp. 11–12

D.4. Scale and Translation Computation

  • For symmetric parts, CAPTRA assumes the symmetry axis aligns with y and uses xz-plane projections with a 2D Umeyama algorithm [27] to compute the axis rotation.Predicted coordinates may differ from ground truth by an additional rotation around the symmetry axis; scale and translation are then computed as for asymmetrical objects.Source: §D.4 · p. 12

E.1. Network Input

  • For synthetic articulated data, the network receives a single-view depth cloud downsampled to 4096 points; for NOCS-REAL275, it crops a ball around the estimated object center and downsamples to 4096 points.The real-data crop radius is 1.2 times the estimated object radius.Source: §E.1 · p. 12

E.2. Training Details

  • Training uses PyTorch and Adam with a learning rate starting at 10^-3 and halving every 20 epochs; convergence takes about 20 rigid-data epochs and 100 articulated-data epochs.Source: §E.2 · p. 12

E.3. Network Architecture

CoordinateNet and RotationNet share a PointNet++ MSG segmentation backbone [23], then use task-specific heads for coordinate and segmentation prediction or rotation regression.

E.3. Network Architecture

  • Both CoordinateNet and RotationNet use a PointNet++ MSG segmentation network as their backbone [23].The architecture also includes set-abstraction and feature-propagation layers.Source: §E.3 · p. 12
  • CoordinateNet branches into a coordinate head with MLP([128, 3P]) and Sigmoid and a segmentation head with FC([P + 1]) and Softmax.LeakyReLU and group normalization are used for each fully connected layer in the set-abstraction and feature-propagation layers.Source: §E.3 · p. 12
  • RotationNet uses the shared backbone followed by MLP([512, 512, 256, 6P]) for its output.LeakyReLU and group normalization are used for each fully connected layer in the set-abstraction and feature-propagation layers.Source: §E.3 · p. 12

F. SAPIEN Articulated Objects Data Generation and Statistics

The section describes a synthetic articulated-object tracking dataset rendered with SAPIEN and summarizes its category definitions, splits, test-sequence motion, and training pose perturbations.

F. SAPIEN Articulated Objects Data Generation and Statistics

  • The SAPIEN dataset statistics cover part definitions, train-test splits, average joint-state changes across 100-frame test sequences, and training pose-perturbation noise; trajectories vary both global pose and joint states.Table 7 summarizes the synthetic articulated-object dataset statistics.Source: F. SAPIEN Articulated Objects Data… · p. 12, Table 7 · p. 13
  • Training noise varies by category according to pose-estimation difficulty, with larger perturbations used for difficult-to-predict thin glasses temples.The larger perturbations are intended to handle larger prediction errors during tracking.Source: F. SAPIEN Articulated Objects Data… · p. 12

G. Experiment Details of Real Drawers under Robot-Object Interaction · H. Discussions on not Using RANSAC · I. Additional Results

The real-drawer experiment trains on synthetic depth data of a simulated robot pushing drawers. The discussion attributes avoiding RANSAC to pose canonicalization and reports only small gains from adding RANSAC to scale and translation computation.

G. Experiment Details of Real Drawers under Robot-Object Interaction

H. Discussions on not Using RANSAC

  • Pose canonicalization simplifies rotation regression and reduces coordinate-prediction noise, allowing the pipeline to avoid RANSAC.The passage notes that orthogonal Procrustes rotation estimation in coordinate-based approaches is sensitive to outliers.Source: H. Discussions on not Using RANSAC · p. 13
  • For NOCS-REAL275 rigid objects, adding RANSAC improves 5°5cm accuracy by 0.86% and mIoU by 1.85%.RANSAC is incorporated only into scale and translation computation.Source: H. Discussions on not Using RANSAC · p. 13
  • For SAPIEN articulated objects, adding RANSAC improves 5°5cm accuracy by 0.06% and mIoU by 0.06%.RANSAC is incorporated only into scale and translation computation.Source: H. Discussions on not Using RANSAC · p. 13
  • Removing RANSAC from C-CoordinateNet drops 5°5cm accuracy by 3.22% and mIoU by 4.37%.The passage attributes these drops to rotation sensitivity to outliers.Source: H. Discussions on not Using RANSAC · p. 13

I.1. Per-Category Results for Rigid Objects … I.4. Additional Visualization

The supplementary results report category-specific rigid-object performance, per-part articulated-object tracking, robustness to pose errors, and additional visual comparisons. They also identify camera-category segmentation as a limitation of the depth-only approach.

I.1. Per-Category Results for Rigid Objects

  • Our method fails on the camera category but outperforms prior state-of-the-art methods under most metrics on all other NOCS-REAL275 categories.Table 8 summarizes per-category rigid-object pose-tracking results.Source: Table 8 · p. 13, §I.1 · p. 13

I.2. Per-Part, Per-Category Results for Articulated Object Tracking

  • On held-out SAPIEN instances, our method achieves the best overall articulated-object tracking performance and outperforms baselines and ablated versions in most cases.Table 10 reports per-part, per-category results on the synthetic dataset.Source: Table 10 · p. 14, §I.2 · p. 14

I.1. Per-Category Results for Rigid Objects

  • Camera performance improves significantly with better segmentation predictions, addressing failures caused by noisy black camera instances that CoordinateNet cannot segment reliably.The authors test Mask-RCNN detections from [29] and ground-truth masks; for multiple camera RoIs, they select the one with greatest overlap with a box projected from the previous pose.Source: Table 9 · p. 13, §I.1 · p. 14

I.3. Robustness to Pose Errors

  • For articulated objects, increased pose errors cause less than a 1-point drop on both 5◦5cm and mIoU metrics.The tests vary initialization errors and errors accumulated across previous-frame predictions.Source: §I.3 · p. 14, Table 11 · p. 14
  • For rigid objects, performance degrades gracefully as pose errors increase, and the method is more robust to noise than 6-PACK.The tests increase initialization error (Init.×1/×2) or add errors to every previous prediction (All.×1/×2).Source: Table 11 · p. 14, §I.3 · p. 14

I.4. Additional Visualization

  • Additional visual tracking results compare our method with oracle ANCSH on SAPIEN, 6-PACK on NOCS-REAL275, and oracle ANCSH on real BMVC laptop trajectories.For BMVC, both methods are trained only on synthetic SAPIEN data and tested on real data without fine-tuning; supplementary videos provide further results.Source: §I.4 · p. 14, Figure 6 · p. 15, Figure 7 · p. 16

Related papers

Loading 2104.03437v2…