Source-linked AI summary
Deep Learning for Spacecraft Pose Estimation from Photorealistic Rendering
Pedro F. Proenca, Yang Gao
TL;DR
The paper addresses monocular 6D pose estimation for known uncooperative spacecraft under harsh lighting and Earth-textured backgrounds. It introduces the URSO photorealistic simulator and an orientation soft-classification framework, achieving 3rd place on ESA’s synthetic test set and 2nd place on its real test set.
Problem
Precise and robust relative pose estimation is needed for close-proximity spacecraft operations involving known or unknown non-cooperative targets under harsh lighting and highly textured Earth backgrounds.
Method
The paper combines the URSO Unreal Engine 4 simulator for photorealistic labeled spacecraft imagery with soft orientation classification that represents pose orientations as probability distributions over a discretized output space.
Results
The best model achieved 3rd place on the ESA synthetic test set and 2nd place on the real test set, while soft orientation classification outperformed direct orientation regression.
Takeaways & Limitations
Soft orientation classification provides better orientation estimates than direct regression and supports modelling uncertainty from orientation ambiguity, while URSO-trained models can generalize qualitatively to real space images.
Abstract
from arXiv · showhide
On-orbit proximity operations in space rendezvous, docking and debris removal require precise and robust 6D pose estimation under a wide range of lighting conditions and against highly textured background, i.e., the Earth. This paper investigates leveraging deep learning and photorealistic rendering for monocular pose estimation of known uncooperative spacecrafts. We first present a simulator built on Unreal Engine 4, named URSO, to generate labeled images of spacecrafts orbiting the Earth, which can be used to train and evaluate neural networks. Secondly, we propose a deep learning framework for pose estimation based on orientation soft classification, which allows modelling orientation ambiguity as a mixture of Gaussians. This framework was evaluated both on URSO datasets and the ESA pose estimation challenge. In this competition, our best model achieved 3rd place on the synthetic test set and 2nd place on the real test set. Moreover, our results show the impact of several architectural and training aspects, and we demonstrate qualitatively how models learned on URSO datasets can perform on real images from space.
I. INTRODUCTION
The paper addresses monocular 6D pose estimation for known uncooperative spacecraft under harsh lighting and textured Earth backgrounds. It introduces photorealistic simulation and an orientation soft-classification framework, evaluating both synthetic and real-data performance.
- Motivation: Precise relative pose and trajectory estimation supports formation flying, rendezvous, docking, servicing, and debris removal under difficult visual conditions.Targets may be cooperative or non-cooperative; the paper focuses on known non-cooperative spacecraft.
- Contributions: URSO uses Unreal Engine 4 to generate photorealistic spacecraft images and depth masks for training and evaluation.The simulator depicts commonly used spacecraft orbiting Earth.
- Contributions: Orientation soft classification represents labels in a discrete orientation space and supports probabilistic estimation of ambiguous orientations.The framework is designed to model multiple hypotheses rather than relying only on direct regression.
- Findings: Random camera-orientation perturbations are reported as effective against overfitting, while models trained on URSO data qualitatively generalize to real space images.The paper studies architectural and training choices on URSO datasets.
- Findings: The best model placed 3rd on the ESA synthetic test set and 2nd on its real test set.The reported competition results cover both synthetic and real evaluation sets.
III. POSE ESTIMATION FRAMEWORK
The pose-estimation framework uses a ResNet-based architecture with separate location and orientation processing. Location is learned through relative-error regression so loss weights can transfer more readily across datasets.
- Architecture: The architecture retains spatial feature resolution by removing the original global average pooling and final fully connected layer from a pretrained ResNet backbone.The design prioritizes simplicity for an initial ablation study and uses an added strided convolution.
- Location estimation: 3D location is estimated with a regression branch containing two fully connected layers.The branch minimizes relative translation error rather than absolute Euclidean distance.
- Location estimation: Relative-error loss makes tuned loss weights less dependent on translation scale and therefore better able to generalize across datasets.The authors also tested regressing three virtual 3D keypoints to avoid fine-tuning loss weights.
A. Direct Orientation Regression
The paper argues that common quaternion L1 or L2 regression losses do not correctly express angular distance. It therefore experiments with direct geodesic-style quaternion losses, while noting a derivative issue that requires scaling.
- Motivation: Quaternion L1 or L2 regression does not correctly represent angular distance for every orientation representation.Quaternions are non-injective, and hemisphere mapping does not fully solve the geodesic-distance mismatch near the equator.
- Method: The inverse-cosine derivative becomes infinite at x = 1, which the authors address by scaling down x.This is presented as a practical adjustment to the direct angular-loss formulation.
B. Probabilistic Orientation Soft Classification
Probabilistic orientation soft classification encodes each ground-truth orientation as a Gaussian over discrete Euler-angle bins. At inference, bin activations are converted into a quaternion through weighted least squares.
- Training: Each orientation label is encoded as a Gaussian random variable over a 3D histogram of discrete Euler-angle bins.The bins correspond to quaternions, with special handling for Gimbal lock and orientation-space borders.
- Training: The kernel uses normalized angular differences between quaternions to distribute probability mass across orientation bins.This makes the soft assignment reflect angular proximity rather than only a single hard class.
- Training: The variance approximates quantization error, while ∆/M is the quantization step, ∆ controls Gaussian width, and M is the number of bins per Euler-angle dimension.These parameters determine the smoothness and resolution of the orientation target distribution.
- Inference: At test time, activations for bins in one quaternion hemisphere are used to fit a quaternion by weighted least squares.The optimal solution is obtained from the right null space of a matrix constructed from the weighted bin quaternions.
- Inference: The quaternion fitting solution follows an approach also employed in prior work.The passage explicitly identifies the prior-work use of this solution.
C. Multimodal Orientation Estimation
The method extracts multiple orientation hypotheses by fitting a Gaussian mixture to multimodal network outputs caused by ambiguous spacecraft views.
- Ambiguous views create one-to-many mappings, so soft-assignment training produces multimodal orientation distributions.
- An Expectation-Maximization framework fits a Gaussian Mixture model to extract multiple orientation hypotheses from the network output.
- During the M step, each model mean is obtained using weights defined by the model membership probabilities.
- The model means are initialized from the strongest post-suppression activations, and the number of mixture components increases until log-likelihood improvement falls below a threshold.
IV. URSO: UNREAL RENDERED SPACECRAFT ON ORBIT
URSO uses Unreal Engine 4 to render spacecraft scenes around Earth, randomizing viewpoints and poses while providing RGB images and depth maps for labeled datasets.
- URSO leverages UE4 features including physically based materials, bloom, and lens flare to render realistic spacecraft images.
- The simulator randomizes Earth rotation, camera orientation, target pose, viewpoint, and target placement within a 10–40 m operating range.
- Each viewpoint provides an RGB image and depth map through the UnrealCV interface.
- Figure 3 includes warped images from camera-orientation perturbations, sim-to-real post-processing examples, and real Soyuz images with estimated-pose overlays.
- Images are rendered at 1080×960 pixels using a virtual camera with a 90° horizontal field of view and auto-exposure.
V. DATA AUGMENTATION AND SIM-TO-REAL TRANSFER
The augmentation strategy uses carefully selected image transformations, including small camera-orientation perturbations implemented by warping images.
- Cropping and flipping require caution because they can alter object appearance and camera intrinsic parameters embedded in the network.
- Random in-plane rotation is used only for the ESA and Stanford dataset because spacecraft may leave the image bounds under other aspect ratios.
- Small random camera-orientation perturbations are applied by warping images to augment training data.
VI. EXPERIMENTS
Experiments evaluate URSO datasets and the SPEED benchmark using synthetic and real imagery, with performance assessed through location, angular, and ESA challenge errors.
- The SPEED benchmark contains 300 real test images, 2,998 synthetic test images, 12,000 synthetic training images, and only 5 real training images.
- All images in the SPEED benchmark are grayscale.
- URSO experiments use one dragon dataset and two Soyuz datasets, each containing 5,000 images with separate testing and validation subsets.
- The Soyuz datasets use operating ranges of 10–20 m for soyuz_easy and 10–40 m for soyuz_hard.
- Performance is reported using mean absolute location error, mean angular error, and ESA Error.
A. Implementation and Training Details
The networks were trained with SGD and a pretrained Mask R-CNN backbone, while learning-rate scheduling depended on convergence and orientation-estimation configuration.
- Training used SGD with momentum 0.9, weight decay 0.0001, and batches of 4 images.
- Models initialized from Mask R-CNN backbones trained on COCO because the approach uses high-resolution images.
- Step-decay learning rates were scheduled according to convergence, which depended on orientation estimation, bin count, augmentation, and dataset configuration.
B. Results
Experiments examine orientation estimation, architecture, augmentation, and dataset difficulty. Soft classification outperformed regression, multimodal estimation represented ambiguous orientations, and the best challenge submission placed second on real data and third on synthetic data.
- Orientation estimation: The Gaussian-tail scale parameter acted as a regularizer: values that were too small caused overfitting, while values that were too large reduced precision and caused underfitting.
- Orientation estimation: Increasing bins per orientation dimension improved precision but caused cubic growth in network parameters and could increase overfitting.Bins become less frequently activated during training as their number increases.
- Orientation estimation: Soft-classification orientation estimation outperformed three regression alternatives and showed less overfitting on soyuz_easy.The alternatives included two orientation-loss regressors and 3D-point regression.
- Orientation estimation: The multimodal framework learned two orientation modes for a symmetrical marker, enabling evaluation using the best of two hypotheses.Top-1 used the single-hypothesis method, whereas Top-2 selected the lower-error hypothesis from two EM-estimated modes.
- Architecture: Performance changed substantially between 8 and 128 bottleneck convolutional filters, while gains beyond 128 imposed a large memory cost.Performance was not very sensitive to the size of the head input layers.
- Architecture: ResNet-50 outperformed shallower networks, whereas adding more layers produced little further improvement; orientation was sensitive to input resolution, unlike localization.
- Training and datasets: Rotation perturbations effectively augmented soyuz_easy, and sim-to-real augmentation was essential for applying URSO-trained models to real footage.The augmentation particularly addressed lighting changes.
- Training and datasets: SPEED was easiest under better lighting, while dragon_hard was most challenging because of viewpoint ambiguity.
C. Conclusion and Future Work
The paper concludes that soft-classification orientation estimation improves spacecraft pose estimation over direct regression while providing uncertainty modelling. Future work targets temporal tracking, unknown-geometry targets, and a more scalable architecture.
- C. Conclusion and Future Work: Soft classification gives better orientation-estimation results than direct regression and provides a means to model uncertainty.The uncertainty representation can support filtering when pose estimates are available over a temporal sequence.
- C. Conclusion and Future Work: Testing includes errors distributed by object distance and qualitative success and failure cases under ambiguous views and poor lighting.The figure descriptions also show good results under challenging conditions, but do not report corresponding numerical outcomes.
- C. Conclusion and Future Work: Future work will use Recurrent Neural Networks with video sequences generated by URSO to address tracking.
- C. Conclusion and Future Work: URSO is planned for extension to SLAM to address targets with unknown geometry.
- C. Conclusion and Future Work: The proposed architecture is not scalable in image and orientation resolution.The authors suggest replacing dense connections, such as by pruning last-layer connections, and sharing a backbone across datasets.