PaperlayerView exact revision on arXiv

Source-linked AI summary

Satellite Pose Estimation Challenge: Dataset, Competition Design and Results

Mate Kisantal, Sumant Sharma, Tae Ha Park, Dario Izzo, Marcus Märtens, Simone D'Amico

arXiv:1911.02050v2cs.CV

TL;DR

Reliable monocular pose estimation for uncooperative satellites is needed for autonomous servicing and debris removal, but scarce annotated spacecraft imagery limits training and comparison. This paper presents the SPEED dataset and SPEC competition design, then analyzes 48 submissions. The analysis identifies target distance and cluttered backgrounds as major difficulties and finds that PnP-based pipelines with separate localization were among the strongest approaches, while real-image performance was generally weaker than synthetic-image performance.

  • Problem

    Spacecraft pose estimation lacks large, accurately annotated datasets and common benchmarks, despite its importance for autonomous servicing and debris removal.

  • Method

    The paper reviews SPEED and SPEC’s competition design, then analyzes submissions and participant survey results to compare approaches and identify sources of task difficulty.

  • Results

    Target distance and cluttered backgrounds were the most significant difficulty factors, while PnP-based keypoint methods and separate localization characterized high-performing pipelines.

  • Takeaways & Limitations

    SPEC submissions significantly outperformed the previous state of the art, but most teams achieved weaker results on real images than on synthetic images.

Abstract

from arXiv · show

Reliable pose estimation of uncooperative satellites is a key technology for enabling future on-orbit servicing and debris removal missions. The Kelvins Satellite Pose Estimation Challenge aims at evaluating and comparing monocular vision-based approaches and pushing the state-of-the-art on this problem. This work is based on the Satellite Pose Estimation Dataset, the first publicly available machine learning set of synthetic and real spacecraft imageries. The choice of dataset reflects one of the unique challenges associated with spaceborne computer vision tasks, namely the lack of spaceborne images to train and validate the developed algorithms. This work briefly reviews the basic properties and the collection process of the dataset which was made publicly available. The competition design, including the definition of performance metrics and the adopted testbed, is also discussed. The main contribution of this paper is the analysis of the submissions of the 48 competitors, which compares the performance of different approaches and uncovers what factors make the satellite pose estimation problem especially challenging.

I. INTRODUCTION

Satellite pose estimation supports autonomous servicing and debris-removal missions, but uncooperative targets and scarce annotated spacecraft imagery make reliable evaluation difficult. SPEC addresses this gap by providing a public dataset and comparing submitted approaches.

  • Autonomous servicing and debris removal require onboard estimates of an uncooperative spacecraft’s relative position and attitude.
  • Spacecraft pose estimation lacks the large, accurately annotated image datasets available for terrestrial computer vision.This scarcity also prevents systematic comparison of competing algorithms.
  • SPEC was organized by SLAB and ESA on the Kelvins platform to address limited data and benchmarking for satellite pose estimation.
  • SPEED combines mostly synthetic imagery with a smaller real-image set collected using a satellite mockup and the TRON facility.The real images provide an indication of submitted methods’ generalization capability.
  • The paper analyzes SPEC submissions to identify task difficulties and compare pose-estimation approaches.It reports target distance and background as main challenges, while PnP-based and separate-detection pipelines performed better.

II. RELATED WORK

Spacecraft pose estimation has evolved from handcrafted feature matching toward deep-learning methods, but progress depends on realistic, labeled imagery. SPEED combines rendered and facility-captured Tango spacecraft images to support this transition.

  • Classical monocular methods extract handcrafted features such as corners, edges, lines, SIFT, SURF, or ORB before iterative pose optimization.
  • Deep neural networks gained prominence through increased GPU resources and large-scale datasets for classification, detection, and object pose estimation.
  • SPEED is the first publicly available machine-learning dataset for spacecraft pose estimation and includes synthetic and real Tango spacecraft imagery.
  • Synthetic SPEED images are rendered with desired ground-truth poses, varied Earth backgrounds, matched illumination, Gaussian blur, and Gaussian white noise.Earth images appear in half of the synthetic images; processing uses σ = 1 blur and σ2 = 0.0022 noise.
  • SPEED synthetic imagery closely matches PRISMA flight-image illumination, with overlapping pixel-intensity histograms and improved rendering over earlier synthetic imagery.

B. Collecting real images with TRON

Real spacecraft imagery was collected in the TRON facility using a full-scale Tango mockup, a robotic camera arm, and controlled illumination. The dataset combines these real images with synthetic imagery, but real-image pose diversity is constrained by the testbed.

  • Real-image collection: TRON used a 1:1 Tango mockup, a seven-degrees-of-freedom robotic arm, LED wall panels, and a xenon short-arc lamp to capture real images.The setup was designed to reproduce diffused Earth-albedo illumination and other relevant imaging conditions.
  • Synthetic–real comparison: Synthetic and real images share ground-truth poses and Earth-albedo direction but differ in spacecraft texture, illumination, and feature eclipse.These discrepancies create a visual gap between the two image sources despite matching pose conditions.
  • Dataset composition: SPEED contains almost 15,000 synthetic images and 300 real images, with only five labeled real images available for training.All images are grayscale and have a resolution of 1920 × 1200 pixels.
  • Dataset geometry: Real-image distances range from 2.8 to 4.7 meters, whereas synthetic-image distances span 3 to 40.5 meters.The narrower real-image range results from physical limitations of the TRON facility and mockup size.
  • Dataset geometry: Real images have restricted orientation and distance diversity because of physical limitations, while synthetic orientations are well distributed across 3D space.This difference affects how broadly each image source samples the pose space.

IV. COMPETITION DESIGN

SPEC was designed to engage a broad community while evaluating pose estimation through a metric aligned with the scientific problem. The competition used accessible participation, rapid leaderboard feedback, and private testing to support sustained and fair comparison.

  • Community engagement: SPEC emphasized broad community engagement by lowering entry barriers, attracting individuals and teams, and maintaining participant involvement.The released dataset was expected to contain sufficient information and well-distributed pose samples for solving the task.
  • Competition metric: The competition metric converts the scientific problem into an optimization problem and must reflect the aspects that matter for practical pose estimation.If the metric can be gamed, high scores may reward solutions with limited practical value.
  • Challenge scope: SPEC focused community efforts on estimating the pose of uncooperative satellites and provided baseline solutions and a competition metric.The competition setup was presented alongside the challenge problem and evaluation design.
  • Competition platform: Kelvins provided a live leaderboard so teams could compare submissions with peers, track progress, and remain motivated over longer intervals.The leaderboard supported continuous feedback during the competition.

B. Competition Metric

The competition metric combines normalized position and orientation errors into a single score averaged over the test set, with sensitivities designed to remain reasonably balanced.

  • Metric design: The leaderboard uses one scalar score combining orientation and position errors rather than separate metrics.The metric was designed to reflect the scientific problem and discourage optimization of only one error type.
  • Position error: Position error is the 2-norm difference between ground-truth and estimated target position vectors in the camera frame.The normalized position error penalizes errors more heavily when the satellite is closer.
  • Orientation error: Orientation error is the angular distance between predicted and true unit quaternions.The quaternion inner product appears in the angular-distance calculation.
  • Aggregate score: Single-image pose error is the 1-norm sum of orientation error and normalized position error, and total error is their average over N test images.The combined score therefore aggregates both error types across the test set.
  • Balance: A 0.1 m translation error was equivalent to 0.7094° orientation error for the test pose distribution, supporting omission of additional scaling factors.The reciprocal comparison reported 1° orientation error as equivalent to 0.141 m translation error.
  • Alternative metrics: Reprojection and average-distance metrics were rejected because their position-orientation sensitivity depends on keypoint choice, with reprojection also unstable near the image plane.The dependence arises because orientation-error sensitivity scales with keypoint distance from the target-body origin.

C. Baseline solutions

The competition supplied deliberately weak, accessible CNN baselines to reduce entry barriers while leaving clear opportunities for incremental improvement.

  • Purpose: Two Python example solutions using Keras and PyTorch were provided so participants could submit an initial result within an hour.The platform also included visualization functions and data loaders for working with samples and pose labels.
  • Architecture and training: The baselines used pre-trained ResNet models with seven linear pose outputs, 224×224 inputs, and simple MSE loss for 20 epochs.The seven outputs represented the pose variables.
  • Weaknesses: MSE loss allowed the camera-boresight distance error to dominate because that output was typically one order of magnitude larger than the others.The baseline outputs were also not normalized.
  • Improvement path: The intentionally simple baselines supported incremental improvements such as changing the loss function or using larger input images.A stronger CNN baseline developed by SLAB during the competition was used for comparison.

V. COMPETITION RESULTS

The 48-team competition substantially improved over the initial baselines, while performance generally remained weaker on real imagery than on synthetic imagery.

  • Participation: 48 teams submitted results, and 20 provided detailed descriptions through a post-competition questionnaire.The analysis compared approaches, described the top four, evaluated performance, and identified difficult samples.
  • Overall performance: The top teams achieved a two-orders-of-magnitude improvement over the initial baseline solutions.The first 20 teams significantly outperformed the initial baseline.
  • Synthetic-to-real performance: Most teams performed worse on the real test set than on the synthetic test set, although three solutions were exceptions.The authors suggest the exceptions may relate to the real imagery’s limited pose distribution.
  • Leading submissions: UniAdelaide won on the synthetic test set, EPFL_cvlab7 was most accurate on real images, and pedro_fairspace was the best non-PnP submission in third place.These three were the only submissions to outperform the SLAB baseline.

B. Approaches of the top 4 competitors

Top-performing pipelines combined deep learning with localization, keypoint or landmark processing, and PnP-based pose recovery, while teams varied in preprocessing and model design.

  • Approaches of the top 4 competitors: UniAdelaide combined 11-landmark triangulation, object-detection bounding boxes, cropped-image landmark regression, and HRNet heatmaps.Its pipeline used predicted landmarks to obtain pose estimates, while another described top approach used EPnP after keypoint regression.
  • Approaches of the top 4 competitors: All surveyed teams used deep learning, either end-to-end or as an intermediate stage, with most framing pose estimation as regression.One team instead treated orientation prediction as a soft classification problem.
  • Approaches of the top 4 competitors: 18 of 20 surveyed teams used data augmentation, including geometric transformations such as camera-axis rotation and zooming.Teams used architectures ranging from pre-trained ResNets, Inception v3, and YOLO to custom models.
  • Approaches of the top 4 competitors: Top-scoring teams localized the satellite before cropping to reduce information loss caused by downscaling high-resolution images.Input resolutions ranged from 224×224 to 960×640 pixels.
  • Approaches of the top 4 competitors: Seven teams reconstructed 3D coordinates for 8–11 keypoints from 10–20 hand-selected images and provided pose labels.The keypoints generally represented satellite-body corners and antenna tips.
  • Approaches of the top 4 competitors: The figures compare direct and PnP-based methods and show the effect of localization before orientation estimation.These comparisons complement the reported prominence of PnP-oriented pipelines among leading approaches.

D. Comparing approaches

The submission analysis compares keypoint matching with direct pose estimation and examines how separate localization changes the pose-estimation pipeline.

  • The analysis compares keypoint matching techniques with pure deep learning approaches and evaluates the effect of a separate localization step.

1) Keypoint matching techniques:

PnP-based keypoint matching substantially outperformed direct pose estimation, while separate localization improved orientation accuracy by enabling image cropping and zooming. The hardest samples combined distant, small satellites with cluttered Earth backgrounds.

  • Keypoint matching techniques: PnP-based solutions significantly outperformed direct pose estimation in both position and orientation among the top 10 teams.
  • Keypoint matching techniques: 0.0328 ± 0.0430 m and 0.0083 ± 0.0269 m were the relative position errors for direct and PnP methods, respectively.
  • Separate localization: Separate localization crops high-resolution images to retain relevant satellite regions, but adds complexity and requires segmentation or bounding-box annotations.
  • Separate localization: Separate localization produced nearly identical position errors but significantly better orientation accuracy than combined localization and pose estimation.
  • Separate localization: 29.66°±46.10° versus 48.03°±49.38° were the mean orientation error and deviation for separate localization and the combined approach, respectively.
  • Difficulty of samples: Earth backgrounds and small targets at large inter-spacecraft distances were the most challenging sample conditions.
  • Competition result: 0.41°±1.50° orientation error and 0.13 ± 0.09 m relative position error were achieved by the winning UNIADELAIDE team.

VI. CONCLUSION AND FUTURE WORK

The challenge benchmarked satellite pose-estimation approaches and analyzed submissions to identify performance drivers. Results favored hybrid DNN-plus-PnP pipelines and separate localization, while exposing gaps in real-image generalization and unknown-object operation.

  • Nearly 50 teams participated in the five-month Satellite Pose Estimation Challenge, which benchmarked different satellite pose-estimation approaches.
  • Target distance and cluttered backgrounds were the most significant factors contributing to sample difficulty.
  • PnP-based keypoint matching with DNN-detected keypoints won the first two places, while DNNs remained central to perception.
  • Separate localization can improve pose accuracy by cropping irrelevant image regions when high-resolution inputs and GPU memory constraints limit input resolution.
  • Performance on real images was relatively poor because their statistical distributions differed from those of the synthetic training images.
  • Future work: Future challenges must better reflect the synthetic-to-real domain gap and address pose estimation for unknown resident space objects.
Loading 1911.02050v2…