PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Deep Learning-Based Monocular Spacecraft Pose Estimation: Current State, Limitations and Prospects

Leo Pauly, Wassim Rharbaoui, Carl Shneider, Arunkumar Rathinam, Vincent Gaudilliere, Djamila Aouada

arXiv:2305.07348v3cs.CVcs.AI

TL;DR

Uncooperative-spacecraft pose estimation supports on-orbit servicing and debris removal, but current datasets and evaluation remain insufficient for deployment. This survey reviews algorithm families, datasets, and deployment limitations; it reports better comparative performance for hybrid approaches in challenge results.

  • Problem

    Uncooperative-spacecraft pose is needed for orbital servicing and debris-removal operations, while current datasets and evaluation procedures remain insufficient for deployment.

    Source: §1 · p. 1, §3.3 · p. 18

  • Method

    The survey reviews hybrid modular and direct end-to-end algorithms, their deployment trade-offs, dataset limitations, and evaluation challenges.

    Source: §2.3 · p. 8, §2.4 · p. 9, §3.3 · p. 18

  • Results

    In KSPEC’19 top-10, hybrid methods had lower mean position errors (0.0083 ± 0.0269 m vs 0.0328±0.0430 m) and orientation errors (1.31 ± 2.24° vs 9.76 ± 18.51°); KSPEC’21 winners used hybrid approaches.

    Source: §2.3 · p. 8

  • Takeaways & Limitations

    Current datasets and evaluation procedures are still insufficient to enable deployment of DL-based spacecraft pose-estimation algorithms in space missions.

    Source: §3.3 · p. 18

    Off-the-shelf models can be computationally expensive, require substantial processing power and memory, and may not be supported by space-system AI accelerators.

    Source: §2.4.1 · p. 9

Abstract

from arXiv · show

Estimating the pose of an uncooperative spacecraft is an important computer vision problem for enabling the deployment of automatic vision-based systems in orbit, with applications ranging from on-orbit servicing to space debris removal. Following the general trend in computer vision, more and more works have been focusing on leveraging Deep Learning (DL) methods to address this problem. However and despite promising research-stage results, major challenges preventing the use of such methods in real-life missions still stand in the way. In particular, the deployment of such computation-intensive algorithms is still under-investigated, while the performance drop when training on synthetic and testing on real images remains to mitigate. The primary goal of this survey is to describe the current DL-based methods for spacecraft pose estimation in a comprehensive manner. The secondary goal is to help define the limitations towards the effective deployment of DL-based spacecraft pose estimation solutions for reliable autonomous vision-based applications. To this end, the survey first summarises the existing algorithms according to two approaches: hybrid modular pipelines and direct end-to-end regression methods. A comparison of algorithms is presented not only in terms of pose accuracy but also with a focus on network architectures and models' sizes keeping potential deployment in mind. Then, current monocular spacecraft pose estimation datasets used to train and test these methods are discussed. The data generation methods: simulators and testbeds, the domain gap and the performance drop between synthetically generated and lab/space collected images and the potential solutions are also discussed. Finally, the paper presents open research questions and future directions in the field, drawing parallels with other computer vision applications.

1. Introduction

Monocular spacecraft pose supports relative navigation for on-orbit servicing and debris-removal operations. This survey reviews deep-learning approaches and their deployment challenges, including domain gaps between training and test imagery.

1. Introduction

  • Knowing the target pose is essential for relative-navigation algorithms to generate real-time trajectories during rendezvous operations.Monocular cameras are favored over active sensors for their simplicity, small size, weight, power requirements, and integration flexibility.Source: §1 · p. 2
  • Target position and orientation relative to a chaser-mounted camera are the pose quantities estimated from a single monocular image.The survey represents position as translation and orientation as rotation.Source: Figure 1 · p. 1, §1 · p. 2
  • Hand-engineered feature approaches perform poorly under variable illumination, low signal-to-noise ratio, and high contrast, while synthetic-to-test image domain gaps remain significant for DL methods.Laboratory test imagery also differs from spaceborne conditions.Source: §1 · p. 2
  • The survey reviews hybrid modular and direct end-to-end methods, datasets, data-generation tools, testbeds, domain-gap mitigation, validation, and future research directions.Its review mainly covers work published through December 2022.Source: §1 · p. 2

2. Algorithms

The survey organizes deep-learning spacecraft pose-estimation algorithms into hybrid modular and direct end-to-end approaches, distinguishing them by how learning and classical vision are combined.

2. Algorithms

  • DL-based spacecraft pose-estimation algorithms broadly fall into hybrid modular and direct end-to-end categories.The survey presents an overview tree of the reviewed algorithms and their branching into these approaches.Source: §2 · p. 3
  • Hybrid modular approaches combine multiple DL models with classical computer-vision methods for spacecraft pose estimation.Source: §2 · p. 3
  • Direct end-to-end approaches use a single DL model trained end-to-end for pose estimation.Source: §2 · p. 3

2.1. Hybrid Modular Approaches

Hybrid modular methods estimate spacecraft pose through localization, keypoint prediction, and pose computation. The survey describes alternative detectors and keypoint formulations, followed by geometric or learned pose-solving options.

2.1. Hybrid Modular Approaches

  • Hybrid pipelines localize and crop the spacecraft, predict 2D projections of predefined 3D keypoints, then compute pose from those correspondences.The 3D keypoints generally come from the spacecraft CAD model; multiview triangulation or Structure from Motion can be used when it is unavailable.Source: §2.1 · p. 3, §2.1.2 · p. 5

2.1.1. Spacecraft Localisation

  • Multi-stage detectors offer generally accurate detections but higher latency and parameter counts, whereas single-stage detectors are lighter and lower-latency.Faster R-CNN and Mask R-CNN are cited as multi-stage examples; YOLO, SSD, and MobileDet as single-stage examples.Source: §2.1.1 · p. 4

2.1.2. Keypoint Prediction

  • Keypoints can be regressed directly, predicted through segmentation or heatmaps, or formulated as bounding-box detections.A segmentation-driven model uses foreground grid cells to contribute keypoint predictions; a shared YOLO-like detector and heatmap backbone is reported at approximately 0.89M parameters.Source: §2.1.2 · pp. 5–6

2.1.3. Pose Computation

  • Pose computation removes outlier keypoints before PnP solving and can refine the estimate by optimizing a geometric loss such as keypoint reprojection error.RANSAC is commonly used for outlier removal; a Pose Inference Network has also replaced PnP with a differentiable learned computation.Source: §2.1.3 · p. 6

2.2. Direct End-to-end Approaches

Direct approaches use one deep-learning model to estimate spacecraft pose from images, with orientation handled through regression or classification-based methods. The surveyed designs range from compact end-to-end networks to multi-task systems addressing domain generalisation.

2.2. Direct End-to-end Approaches

  • Direct approaches regress spacecraft pose from images using one end-to-end DL model, without intermediate stages or additional camera parameters or a 3D spacecraft model.The models are trained using loss functions calculated from pose error; camera parameters are learned during training.Source: §2.2 · p. 6
  • Phisannupawong et al. regress a 7D pose vector with a GoogLeNet-based CNN, but unit-quaternion norm loss fails to capture angular distance and yields larger errors.Their experiments found better performance with a weighted Euclidean-based loss than with an exponential loss.Source: §2.2 · p. 6, Figure 8 · p. 6
  • Sharma et al. classify images into pose labels quantised across four degrees of freedom, but more labels enlarge the softmax layer and predictions need refinement.The degrees of freedom cover two camera-position dimensions on an enclosing sphere, bore-sight rotation, and camera-to-spacecraft distance.Source: §2.2 · p. 6, Figure 9 · p. 7
  • SPN [146] combines spacecraft localisation, orientation-class probabilities, and class-weight regression; quaternion averaging refines attitude, while Gauss–Newton estimates position.The architecture uses three branches: a bounding-box output, orientation classification, and orientation-class weighting.Source: §2.2 · pp. 6–7, Figure 10 · p. 7
  • URSONet regresses position and uses soft-assignment classification for continuous orientation estimation.Ground-truth orientation labels are encoded as Gaussian random variables in the discrete output space.Source: §2.2 · p. 7, Figure 11 · p. 8
  • 2.2M to 7.4M parameters made Mobile-URSONet 13 times smaller than URSONet without considerable performance degradation.It replaces the ResNet backbone with MobileNetv2 and reduces the sub-branches from two fully connected layers to one.Source: §2.2 · p. 7
  • SPNv2 [110] uses multi-task learning to support domain generalisation and proposes onboard online domain refinement using unlabelled target images.Its smallest 3.8M-parameter variant performs comparably to the 52.5M-parameter best-performing variant on SPEED+ synthetic data.Source: §2.2 · p. 7
  • Garcia et al. separate translation and orientation CNN modules, while E-PoseNet uses an SE(2)-equivariant ResNet18 backbone.E-PoseNet's equivariant features encode geometric information, and its rotation-equivariant ResNet reduces model size relative to a regular ResNet for the same feature size.Source: §2.2 · pp. 7–8, Figure 12 · p. 8

2.3. Algorithm Comparison

The survey compares hybrid and direct algorithms by reported pose errors and model parameter counts, while noting that some counts are estimated. Challenge results favor hybrid approaches, and the comparison highlights accuracy–size trade-offs relevant to edge deployment.

2.3. Algorithm Comparison

  • The survey compares mean position and orientation errors and model parameter counts, using SPEED synthetic-test results where available and estimating some unreported parameter counts.Figure 13 plots algorithm parameter counts against position and orientation errors.Source: §2.3 · p. 8, Figure 13 · p. 9
  • Edge deployment requires balancing model size against performance; the survey identifies algorithms [93, 120, 81] and SLAB Baseline [115] as offering a good trade-off.Hybrid pipelines allow stages to be improved independently, whereas direct models must be retrained to incorporate changes such as camera parameters.Source: §2.3 · p. 8
  • 0.0083 ± 0.0269 m and 1.31 ± 2.24° for hybrid methods, versus 0.0328±0.0430 m and 9.76 ± 18.51° for direct methods among KSPEC’19 top-10 methods; both KSPEC’21 winning streams also used hybrid methods.These results are reported as evidence that hybrid approaches performed comparatively better in the challenges.Source: §2.3 · p. 8

2.4. Limitations

The survey identifies deployability, explainability, and robustness to illumination as unresolved limitations of DL-based spacecraft pose estimation. These constraints remain relevant to implementation in real space missions.

2.4.1. Deployability

  • Only a small fraction of algorithms are evaluated on space-deployment edge systems, and reports often omit latency, inference time, memory, power, and computational cost.Those factors matter for assessing deployment on resource-constrained space systems.Source: §2.4.1 · p. 9
  • Off-the-shelf models can be computationally expensive and require unsupported layers, making them potentially unsuitable for space hardware such as FPGA-based accelerators.The survey calls for architectures customised for space applications and hardware.Source: §2.4.1 · p. 9

2.4.2. Explainability

  • Direct models are difficult to interpret, while hybrid pipelines are more interpretable but still lack reasoning and uncertainty-modelling capabilities.The paper frames explainability as important for trust, safety, error analysis, and troubleshooting in space missions.Source: §2.4.2 · p. 9

2.4.3. Robustness to Illumination Conditions

  • Even the best methods in KSPEC’21 performed poorly under extreme lighting, where shadows, reflections, and sun glare can obscure spacecraft features.Figure 14 shows the three worst predictions on lightbox and sunlamp images.Source: Figure 14 · p. 9, §2.4.3 · p. 10

3. Datasets

The survey reviews spacecraft pose datasets, image-generation tools, and testbeds, then discusses approaches to the domain gap across synthetic, laboratory, and space imagery. It also describes data-level and algorithm-level strategies for addressing that gap.

3. Datasets

  • Public spaceborne image datasets are lacking, so rendering tools and testbeds are used to generate data and validate algorithms across selected targets and scenarios.Rendering tools can generate annotated images for user-defined applications and vary parameters such as camera models and orbital lighting.Source: §3 · p. 14

3.1. Summary of Datasets, Simulators & Testbeds

  • Available spacecraft datasets contain about 10^4 to 10^5 images, and hybrid methods need secondary annotations beyond 6DoF pose labels.The survey notes that target size constrains operating range; secondary labels such as keypoints can require recovering 3D locations from pose labels, while generated annotations cannot replace properly calibrated labels.Source: §3.1 · pp. 14–15
  • Rendering tools include general-purpose and space-focused simulators, but no tool is considered a de facto standard; laboratory testbeds simulate motion and lighting and require careful calibration for accurate pose labels.The survey describes testbeds using robotic manipulators, lighting equipment, and motion-capture systems.Source: §3.1 · pp. 15–16

3.2. Bridging the Domain Gap

  • Domain gaps can cause performance drops between synthetic training data and real images, and can persist between laboratory and space imagery.The survey groups responses into data-level methods that diversify training images and algorithm-level methods that reduce domain-dependent features.Source: §3.2 · p. 16

3.2.1. Data Augmentation

  • Data augmentation varies existing images through pixel-wise changes such as blurring or contrast shifts and spatial changes such as rotation or scaling.Spatial augmentation also requires changing pose labels; augmentation can be counterproductive, and Random Erase was reported to reduce pose-estimation performance [110].Source: §3.2.1 · p. 16

3.2.2. Domain Randomization

  • Domain randomization trains on sufficiently varied source data so the target domain appears as another variation; spacecraft texture has been randomized using neural style transfer [115].It is presented as a particular case of data augmentation that seeks broad variation rather than a context-specific set of transformations.Source: §3.2.2 · p. 16

3.2.3. Multi-Task Learning

  • Multi-task learning shares features across related tasks; in an EfficientPose variant, enabling all task heads gave the best performance, though the segmentation head improved it only slightly.Choosing relevant auxiliary tasks remains a challenge [150].Source: §3.2.3 · pp. 17–18

3.2.4. Domain-Adversarial Learning

  • Domain-adversarial learning attaches a classifier to distinguish source from target images and maximizes its loss during training to encourage domain-invariant, pose-discriminative features.The less the classifier can distinguish the domains, the more domain-invariant the model becomes.Source: §3.2.4 · p. 18

3.3. Limitations

The survey identifies synthetic-image realism and evaluation across domains as limitations for deployment. Its discussion highlights a substantial synthetic-to-laboratory performance gap and cautions that laboratory performance does not establish performance in space.

3.3. Limitations

  • Current datasets and evaluation procedures remain insufficient to enable deployment of DL-based spacecraft pose-estimation algorithms in space missions.The survey identifies limitations in synthetic-data realism and algorithm evaluation.Source: §3.3 · p. 18

3.3.1. Realism of Synthetically Generated Datasets

  • Realistic synthetic space imagery remains an open challenge because rendering must reproduce light interactions in an environment lacking reference points and an atmosphere.The survey calls for graphics techniques tailored to space environments.Source: §3.3.1 · p. 18

3.3.2. Algorithm Evaluation

  • The best pose scores differed by one order of magnitude between the 2019 synthetic-test and 2021 laboratory-test editions of ESA’s challenge [71].Success on laboratory images does not guarantee the same performance on space images because the environments also differ.Source: §3.3.2 · pp. 18–19

4. Future Research Directions

The survey identifies open directions in algorithm development and data generation to address remaining barriers to deploying spacecraft pose estimation in space.

4.1. Deployability of Algorithms

  • Space deployment remains under-evaluated on edge hardware; parameter counts offer an incomplete resource proxy, and unsupported operations challenge accelerator compatibility.The survey recommends reporting model size, FLOPS, and latency, and suggests NAS for models suited to space-system accelerators.Source: §4.1 · pp. 19–20

4.2. Explainability of Algorithms

  • Black-box DL makes inference and predictions difficult to explain, especially for direct end-to-end algorithms, motivating XAI methods such as Bayesian deep learning and conformal inference.The survey frames explainability as relevant to reliability and trustworthiness in safety-critical applications.Source: §4.2 · p. 20

4.3. Multi-Modal Spacecraft Pose Estimation

  • Visible-only methods face acquisition challenges such as low light and overexposure, while multimodal spacecraft pose estimation remains sparsely studied.Thermal and time-of-flight or event cameras are proposed as sensors to extend operational range.Source: §4.3 · p. 20

4.4. Generation of More Realistic Synthetic Data

  • Synthetic data is pervasive, and the survey proposes comparing simulator renderings with space imagery to guide more realistic training data.It identifies PANGU as the only such initiative known to the authors and notes physics-informed layers as another simulation-to-real approach.Source: §4.4 · p. 20

4.5. Domain Adaptation

  • Synthetic-to-real performance gaps remain a deployment obstacle, motivating more realistic rendering and domain adaptation approaches.The survey proposes analyzing renderings against space imagery, physics-informed layers, and methods such as self-supervised, multi-task, or adversarial learning.Source: §4.5 · p. 20, §4.4 · p. 20

4.6. Beyond Target-Specific Spacecraft Pose Estimation

  • Target-specific algorithms require a new dataset and retraining for each spacecraft, motivating spacecraft-agnostic methods for unseen objects and debris shapes.The survey points to generic 6D pose estimation methods for unseen objects as a possible route.Source: §4.6 · p. 20

4.7. Multi-Frame Spacecraft Pose Estimation

  • Current methods estimate pose from isolated frames, while consecutive images can provide temporal information for more accurate and consistent estimates.SPARK2 supplies trajectories, and ChiNet uses LSTM units; the survey also points to video-based 6 DoF methods from computer vision.Source: §4.7 · p. 20

5. Conclusions

The conclusion reviews DL-based monocular spacecraft pose estimation and the limitations affecting its deployment. It summarizes approaches, datasets, domain-gap strategies, and future directions.

5. Conclusions

  • The survey organizes DL-based methods into hybrid modular and direct end-to-end regression approaches.It compares algorithm performance and network architecture size to inform deployability.Source: §5 · pp. 20–21
  • The survey identifies algorithm-design and dataset limitations that remain to be addressed before deployment in actual space scenarios.Its future directions aim at spacecraft pose estimation algorithms deployable in real space missions.Source: §5 · pp. 20–21
  • It discusses datasets for training and validation or testing, along with data-generation methods, simulators, testbeds, and strategies for bridging the domain gap.These topics are presented as part of the survey’s review of limitations and future directions.Source: §5 · pp. 20–21

A.1. Publicly available algorithm implementations

The appendix lists a publicly available algorithm-implementation resource.

A.1. Publicly available algorithm implementations

  • The passage lists a pose-segmentation PDF at an indico.esa.int URL and associates the link with bibliography marker, without establishing it as an algorithm implementation resource.The URL shown is https://indico.esa.int/event/319/attachments/3561/4754/pose_gerard_segmentation.pdf.Source: §A.1 · p. 21

A.2. Links to the publicly available datasets

The appendix lists a publicly available spacecraft dataset resource.

A.2. Links to the publicly available datasets

  • URSO-OrViS is listed as a publicly available dataset at Zenodo record 3279632.The passage supplies the link https://zenodo.org/record/3279632.Source: §A.2 · p. 21
Loading 2305.07348v3…