PaperlayerView exact revision on arXiv

Source-linked AI summary

se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains

Bowen Wen, Chaitanya Mitash, Baozhang Ren, Kostas E. Bekris

arXiv:2007.13866v1cs.CVcs.GRcs.LGcs.ROeess.IV

TL;DR

Long-term 6D pose tracking for robot manipulation is challenged by occlusions, scarce real annotated data, and accumulated drift. The paper proposes se(3)-TrackNet, which learns residual pose updates from current RGB-D observations and synthetic renderings using disentangled features and Lie Algebra representations. Experiments report robust tracking on benchmarks, including without re-initialization, with approximately 90.9Hz operation.

  • Problem

    Real-world 6D pose annotations are difficult to acquire, while long-term tracking must handle occlusions and accumulated error drift.

  • Method

    se(3)-TrackNet learns residual relative poses between current RGB-D observations and synthetic renderings conditioned on previous estimates, using Lie Algebra and physically plausible domain-randomized synthetic training.

  • Results

    The method achieves state-of-the-art YCB-Video tracking without re-initialization and operates at approximately 90.9Hz.

  • Takeaways & Limitations

    Synthetic-only training can transfer robustly to real-world 6D pose tracking, including manipulation settings with large occlusions and sudden re-orientations.

Abstract

from arXiv · show

Tracking the 6D pose of objects in video sequences is important for robot manipulation. This task, however, introduces multiple challenges: (i) robot manipulation involves significant occlusions; (ii) data and annotations are troublesome and difficult to collect for 6D poses, which complicates machine learning solutions, and (iii) incremental error drift often accumulates in long term tracking to necessitate re-initialization of the object's pose. This work proposes a data-driven optimization approach for long-term, 6D pose tracking. It aims to identify the optimal relative pose given the current RGB-D observation and a synthetic image conditioned on the previous best estimate and the object's model. The key contribution in this context is a novel neural network architecture, which appropriately disentangles the feature encoding to help reduce domain shift, and an effective 3D orientation representation via Lie Algebra. Consequently, even when the network is trained only with synthetic data can work effectively over real images. Comprehensive experiments over benchmarks - existing ones as well as a new dataset with significant occlusions related to object manipulation - show that the proposed approach achieves consistently robust estimates and outperforms alternatives, even though they have been trained with real images. The approach is also the most computationally efficient among the alternatives and achieves a tracking frequency of 90.9Hz.

I. INTRODUCTION

Long-term 6D pose tracking supports robot manipulation but remains difficult under occlusion, limited real training data, and domain differences. se(3)-TrackNet addresses these challenges with synthetic-data training, disentangled feature encoding, Lie Algebra orientation representation, and strong tracking performance.

  • Temporal tracking can improve pose-estimation speed and quality, but data-driven methods require difficult-to-acquire real-world 6D pose annotations.
  • se(3)-TrackNet predicts relative pose between the current observation and a synthetic rendering conditioned on the previous estimate.
  • Feature-encoding disentanglement is designed to improve transfer from synthetic training data to real images.
  • Lie Algebra represents 3D orientations for learning residual pose transforms, while domain-randomized synthetic training reduces manual data collection and labeling.
  • 90.9Hz tracking and state-of-the-art YCB-Video results without re-initialization support real-time robot-manipulation scenarios.

III. APPROACH

The approach formulates 6D object-pose tracking over RGB-D sequences as a data-driven optimization problem. It uses the object’s CAD model, an initial pose, prior observations, and the current observation, with a neural network learning the optimization cost from synthetic data.

  • The objective is to compute an object’s 6D pose Tt ∈ SE(3) at any time t > 0.
  • The tracking setup requires a 3D CAD model, an initial pose T0 ∈ SE(3), and RGB-D observations from previous timestamps plus the current frame.
  • The method performs data-driven pose optimization over RGB-D sequences by learning its cost function with a neural network trained only on synthetically generated data.

A. Tracking on SE(3) Manifolds with Residuals

The method formulates pose tracking as learned residual optimization between the current observation and a rendering conditioned on the previous pose. It uses feature discrepancies, local SE(3) parameterization, and a neural network to estimate relative transforms without explicit hand-crafted optimization.

  • The cost function measures feature discrepancy between images associated with candidate object poses using a robust loss and feature extractor.The features may be pixel intensities, point-to-point discrepancies, designed features, or combinations.
  • The optimization seeks a relative transform that moves the previous pose estimate to the pose captured by the current RGB-D observation.The object-model rendering is evaluated at the transformed previous pose.
  • Taylor expansion locally approximates rendered-image features with a Jacobian around the previous pose, using ξ ∈ se(3) as the tangent-space parameterization.The exponential mapping connects the tangent-space representation to the SE(3) transformation.
  • Explicit iterative optimization requires differentiable features, a formalized cost function, robust-loss selection, and modality-weighting hyperparameters.These requirements become especially non-trivial when combining modalities such as RGB-D.
  • The proposed neural network implicitly calibrates residual features and predicts the relative transform directly in the se(3) tangent space.This replaces explicit Jacobian-based optimization with learned residual calibration.

B. Neural Network Design

The network separates feature encoding for the rendered previous pose and current observation to support synthetic-only training with real test images. Its design combines Lie-algebra pose prediction with physically plausible synthetic-data generation.

  • Neural architecture: The network processes rendered previous-pose images and current RGB-D observations through separate, unshared feature encoders before predicting relative pose.Translation and rotation are predicted through decoupled branches.
  • Neural architecture: At training time both inputs are synthetic, whereas at test time the current-observation branch receives a real sensor image.This asymmetric train-test setup motivates disentangled feature encoding.
  • Neural architecture: Feature encoding disentanglement reduces the domain gap by allowing the synthetic-rendering and real-observation representations to be handled separately.A shared extractor was reported not to generalize when training used exclusively synthetic data.
  • Pose representation: The relative transform is represented as Δξ=(t,w)^T∈se(3), with translation and rotation trained in separate branches using an L2 loss.The current pose is updated by applying exp(Δξ) to the previous pose.
  • Synthetic data generation: Physically Plausible Domain Randomization initializes random object poses, then uses physics simulation to render stable, penetration-free configurations.This avoids depth bias from inter-object penetration and is intended to align synthetic and real domains.

C. Synthetic Data Generation via PPDR

PPDR combines domain randomization with physically consistent simulation to generate diverse synthetic training data that better supports generalization to real-world domains. The pipeline also corrupts synthetic depth and aligns real depth to the synthetic domain.

  • C. Synthetic Data Generation via PPDR: Domain randomization is used to help models trained on simulated variability generalize to real-world data.Prior randomization changes scene attributes such as object number, poses, textures, and lighting.
  • C. Synthetic Data Generation via PPDR: PPDR combines randomized scene properties with physics simulation so synthetic scenes remain diverse while avoiding implausible object collisions.Objects are initialized with possible collisions, then simulated so they separate or fall onto the table; lighting, object count, and distractor textures are randomized.
  • C. Synthetic Data Generation via PPDR: Synthetic training pairs are generated from a perturbed previous pose and a current synthetic rendering of the target object.The perturbed translation and Lie-algebra rotation use uniformly sampled directions with Gaussian-distributed magnitudes.
  • C. Synthetic Data Generation via PPDR: Depth-domain alignment adds Gaussian noise and randomly invalidates valid depth pixels during training, while inference filters real depth and fills holes.These augmentations are intended to make synthetic depth resemble corrupted commercial sensor data and align real depth with the synthetic domain.

IV. EXPERIMENTS

The experiments train and evaluate the approach using synthetic image pairs and pose perturbations, with fixed preprocessing, optimization, and rendering procedures. Training uses 200k image pairs and controlled RGB-D augmentations.

  • IV. EXPERIMENTS: The evaluation compares the method with state-of-the-art 6D pose tracking and single-image pose estimation methods across diverse objects and manipulation scenarios.Experiments include both quantitative and qualitative evaluations while using only synthetic training data.
  • IV. EXPERIMENTS: 200k image pairs are used to train the network, with pose perturbations sampled using σ_t=2 cm and σ_w=0.262 rad (15°).The network is trained for 300 epochs with Adam, batch size 200, and learning-rate reductions at epochs 100 and 200.
  • IV. EXPERIMENTS: RGB-D inputs are resized to 176 × 176 before network processing.Rendering of the previous-frame object image is implemented in C++ OpenGL for both training and inference.

A. Datasets

The evaluation uses YCB-Video and a new YCBInEOAT benchmark designed for robotic manipulation. The datasets provide RGB-D sequences with annotated 6D object poses across tabletop and end-effector settings.

  • YCB-Video: YCB-Video contains 92 RGB-D sequences covering 21 YCB objects, with ground-truth 6D poses annotated in every frame.Evaluation reports AUC results on keyframes from 12 test sequences using ADD and ADD-S metrics.
  • YCB-Video: The proposed method uses no real pose-annotated training or validation data for YCB-Video and trains only on synthetic data.The dataset includes real-world pose annotations, but they are not used by the proposed approach.
  • YCBInEOAT: YCBInEOAT introduces robotic manipulation sequences using a vacuum gripper, Robotiq 2F-85 gripper, and Yale T42 Hand.The dataset covers pick-and-place, within-hand manipulation, and hand-off between arms to placement.
  • YCBInEOAT: YCBInEOAT records RGB-D images at 20 to 30 Hz with an Azure Kinect mounted statically on the robot.Five YCB objects are used, and ground-truth camera-frame poses are manually annotated for each video frame.

B. Results on YCB-Video

On YCB-Video, the method is evaluated against state-of-the-art tracking and pose estimation approaches under different initialization and re-initialization settings. It achieves strong ADD and ADD-S results while avoiding costly re-initialization and supporting fast operation.

  • B. Results on YCB-Video: Re-initialization can interrupt planning and control because 6D pose detection is slower and may introduce new errors.This motivates evaluating tracking methods that maintain pose estimates without repeated re-detection.
  • B. Results on YCB-Video: The proposed approach significantly outperforms competing methods on ADD and achieves the highest ADD-S success rate under both initialization settings.The comparison includes ground-truth initialization and initialization from PoseCNN, with and without re-initialization in the latter setting.
  • B. Results on YCB-Video: Qualitative tracking examples cover a large clamp in YCB-Video and a bleach-cleanser manipulated by a vacuum gripper in YCBInEOAT.These examples connect tabletop benchmark tracking with object manipulation scenarios.

C. Results on YCBInEOAT-Dataset

The YCBInEOAT evaluation tests tracking from a ground-truth first-frame pose without re-initialization, including challenging manipulation motions. Qualitative results indicate that the proposed approach maintains reliable long-term pose estimates under these conditions.

  • Pose is initialized with ground truth in the first frame, and no re-initialization or forward kinematics is used.This isolates tracking quality on the YCBInEOAT benchmark.
  • The benchmark includes abrupt motions, extreme rotations, and slippage within the end-effector during vacuum-gripper pick-and-place manipulation.
  • The proposed approach provides sufficiently robust long-term pose estimation until manipulation ends despite these challenges.

D. Ablation Study

The ablation study evaluates domain randomization, depth, encoder sharing, rotation representation, and loss design using ground-truth initialization without re-initialization. Results show that depth, separate encoders, Lie Algebra-based rotation, and the selected loss are important to performance.

  • 94.71 ADD and 96.93 ADD-S are achieved by the proposed configuration, compared with 91.88 and 95.76 without physics simulation.
  • Removing depth reduces performance to 75.65 ADD and 87.22 ADD-S, while sharing the encoders reduces both metrics to 0.28.
  • Quaternion rotation achieves 93.58 ADD and 96.39 ADD-S, whereas Shape-Match Loss achieves 1.93 ADD and 5.48 ADD-S.The quaternion variant uses an L2 loss, while Shape-Match Loss loses track very early in this setting.

V. CONCLUSION

The conclusion presents se(3)-TrackNet as an efficient, robust long-term 6D pose tracker trained on synthetic data and transferred to real-world data. It reports approximately 90.90 fps and robustness to occlusions and sudden re-orientations, while requiring an object CAD model.

  • se(3)-TrackNet combines a neural network architecture with Lie Algebra residual-pose learning to transfer from synthetic training data to real-world tracking.
  • 90.90 fps is reported, and the approach remains robust under large occlusions and sudden re-orientations in the manipulation benchmark.
  • The method requires an object CAD model, while future work targets similar performance for category-level 6D pose tracking.
Loading 2007.13866v1…