Source-linked AI summary
Satellite Pose Estimation with Deep Landmark Regression and Nonlinear Pose Refinement
Bo Chen, Jiewei Cao, Alvaro Parra, Tat-Jun Chin
TL;DR
The paper addresses monocular 6DOF pose estimation for known space-borne objects, a task relevant to docking, debris removal, and communications. It combines deep landmark regression with geometric pose refinement and achieves first place in the Kelvins Pose Estimation Challenge, with low reported orientation and translation errors.
Problem
Monocular 6DOF pose estimation for space-borne objects is important for space operations, while the KPEC setting requires estimating a known satellite's pose from images with varying backgrounds.
Method
The method regresses 2D image landmarks with HRNet, associates them with reconstructed 3D landmarks, and refines the pose through geometric optimisation.
Results
The approach won first place in KPEC, achieving 0.7277 degrees average cross-validation orientation error, 0.0359 metres translation error, and a 0.0094 test-set overall score.
Takeaways & Limitations
Combining high-resolution landmark prediction with robust geometric refinement produced accurate satellite pose estimation and outperformed previous work on the reported benchmarks.
Abstract
from arXiv · showhide
We propose an approach to estimate the 6DOF pose of a satellite, relative to a canonical pose, from a single image. Such a problem is crucial in many space proximity operations, such as docking, debris removal, and inter-spacecraft communications. Our approach combines machine learning and geometric optimisation, by predicting the coordinates of a set of landmarks in the input image, associating the landmarks to their corresponding 3D points on an a priori reconstructed 3D model, then solving for the object pose using non-linear optimisation. Our approach is not only novel for this specific pose estimation task, which helps to further open up a relatively new domain for machine learning and computer vision, but it also demonstrates superior accuracy and won the first place in the recent Kelvins Pose Estimation Challenge organised by the European Space Agency (ESA).
1. Introduction
The paper addresses monocular 6DOF satellite pose estimation for space operations by combining a reconstructed satellite model, deep landmark prediction, and geometric optimisation. In KPEC, the method achieved first place with low cross-validation errors and a 0.0094 test score.
- 6DOF pose estimation supports docking, debris removal, and inter-spacecraft communications, motivating lower-power monocular alternatives to active sensors.
- The KPEC setting uses a known Tango satellite but requires pose estimation across varying backgrounds, object sizes, orientations, and lighting conditions.
- The method reconstructs a satellite 3D model from training images, predicts predefined 2D landmarks, and estimates pose from their 2D-3D correspondences using robust geometric optimisation.
- 0.7277 degrees orientation error and 0.0359 metres translation error were obtained by cross-validation on the KPEC training set.
- 0.0094 overall test score ranked the method first in the Kelvins Pose Estimation Challenge.
2. Related works
Prior pose-estimation methods use handcrafted correspondences, end-to-end prediction, or learned feature matching, while this work combines deep landmark regression with geometric pose estimation for a known spacecraft.
- Keypoint methods: Traditional keypoint methods form 2D-2D or 2D-3D correspondences before nonlinear pose optimisation but can fail under large pose and lighting variations.
- End-to-end learning: End-to-end CNN methods learn image-to-pose mappings, but prior work reports lower accuracy than geometry-based solutions that optimise pose from correspondences.
- Feature learning methods: Deep feature-learning methods predict image locations for fixed 3D keypoints, with differences mainly in architecture and keypoint selection.
- Feature learning methods: The paper selects object-surface keypoints because spacecraft are typically relatively unoccluded and richly textured.
- Feature learning methods: HRNet maintains high-resolution representations throughout the network, a design motivated by their importance in object detection and human pose estimation.
- Spacecraft pose estimation: Unlike model-based spacecraft systems using hand-engineered features and matching, this approach uses a trained deep network to regress 2D landmark coordinates for 2D-3D pose estimation.
- Spacecraft pose estimation: SPN combines bounding-box detection, discretized SO(3) rotation classification, rotation regression, and translation estimation from bounding-box and rotation constraints.
3. Methodology
The pipeline reconstructs a landmark-based 3D satellite model, predicts corresponding 2D landmarks, and refines pose through robust geometric optimisation. It also compares performance against SPN.
- 3.1. Multi-view triangulation: The pipeline reconstructs a satellite 3D structure from manually selected landmarks and training-image correspondences using multi-view triangulation.The implementation uses 11 landmarks, including eight satellite corners and three antenna-end centers.
- 3.2. Object detection and 3.3. Landmark regression: An object detector first predicts a satellite bounding box, after which a landmark regression network predicts image locations for the predefined 3D landmarks.The landmarks’ projected 2D matches cover almost the whole object, supporting bounding-box generation.
- 3.3. Landmark regression: HRNet outputs 11 heatmaps with a fixed one-to-one association between each heatmap and a 3D landmark.The regression model is trained on visible landmark labels and heatmap-based mean squared error.
- 3.4. Pose estimation: The pose estimator solves a robust nonlinear least-squares problem from predicted 2D-3D correspondences using a Huber loss and cheirality constraints.For SPEED images, a RANSAC-style PnP solver provides the initial pose.
- 3.4. Pose estimation: Simulated annealing progressively lowers the Huber and outlier thresholds while removing correspondences with large residuals during Levenberg-Marquardt refinement.The procedure alternates pose refinement and correspondence filtering until minimum thresholds or the iteration limit is reached.
- 3.5. Results: The proposed method achieves superior object-detection and pose-estimation performance compared with SPN, with both rotational and translational errors at least one order of magnitude smaller.Table 1 reports the comparison between SPN and the proposed method.
4. Evaluation
The evaluation measures detection and pose quality with defined geometric and competition metrics, using six-fold cross-validation and an ensemble-based test procedure. Pose refinement and comparison results show improved scores, strong performance against SPN, and first-place KPEC performance.
- Metrics: Rotation, translation, and detection performance are evaluated using rotation error ER, translation error ET, and Intersection over Union (IOU).The KPEC comparison additionally uses rotation score SR, translation score ST, and overall score S.
- Evaluation protocol: Six-fold cross-validation splits the 12,000 training images into six groups, retraining detection and landmark-regression models for each held-out group.Estimated poses use predicted 2D–3D correspondences, RANSAC with a PnP kernel, and subsequent SA-LMPE refinement.
- Evaluation protocol: An ensemble averages six trained models’ landmark heatmaps for each test image before producing final landmark coordinates and poses.The remaining test procedure follows the same pipeline used for pose estimation.
- Comparison results: The proposed method achieves rotational and translational errors at least one order of magnitude smaller than SPN, while also improving object detection and pose estimation.The comparison is reported in Table 1.
- Refinement results: SA-LMPE removed 8495 potential outliers, approximately 0.7 per image, while improving average overall score S from 0.0167 to 0.0117.The refinement used δ = 5, ϵ = 50, δmin = 1, ϵmin = 4, λδ = λϵ = 0.7, and tmax = 10.
5. Conclusion
The paper presents a monocular satellite pose-estimation framework that combines deep landmark prediction with geometric optimisation. It reports first place in KPEC and CV evidence of substantially better performance than previous work on SPEED.
- 5. Conclusion: The framework combines deep neural-network feature learning with geometric optimisation for robust fitting in monocular satellite pose estimation.HRNet predicts 2D landmarks, while SA-LMPE removes inaccurate predictions and refines poses.
- 5. Conclusion: The approach won first place in KPEC, and cross-validation indicates significant outperformance of previous work on the SPEED benchmark.