Monocular Navigation Relative to Unknown Spacecraft Using a Transformer-Aided Kalman Filter

1. INTRODUCTION

• The pipeline combines Spacecraft Odometry Network (SON) measurements with an MSCKF that tracks relative orbit elements, target attitude, and angular velocity for unknown-spacecraft pose estimation.Feature tracks constrain local motion and loop closure supplies longer-baseline relative attitude constraints to limit drift; the filter also uses outlier rejection and covariance inflation.Source: §1 · pp. 1–2
• Because monocular measurements are scale-free, range becomes observable when the servicer provides a known metric camera displacement transverse to the line of sight.The image stream is initialized by an earlier mission phase, such as Angles-Only Navigation (AON).Source: §1 · p. 1
• SPE3R+ expands the dataset to 103 spacecraft models, including 11 test models, with 5,000 higher-resolution images per model, masks, and metric depth.The contribution list describes the expanded dataset as including grayscale images, masks, and depths.Source: §1 · p. 2
• LightGlue fine-tuning raised test-spacecraft RANSAC pose AUC@20° from 4.0 to 17.0 and match recall from 15.7% to 49.1%.On unseen spacecraft under large illumination changes, SON had median rotation errors of 2.6° and translation direction errors of 4.4°, versus 9.9° and 45.2° for the classical baseline; on SHIRT images, its median translation unit direction error was 4–6 times lower. SuperPoint features are matched with LightGlue, fine-tuned for spacecraft imagery.Source: §1 · p. 2
• Median errors were 2.2% of the range in ROE and 3.7° in attitude across a Monte Carlo evaluation of 11 unseen spacecraft and 550 trials.The evaluation included ablations quantifying the value of each pipeline component.Source: §1 · p. 3

2. STATE OF THE ART

• Known-target pose estimation has progressed from retro-reflectors and model-based edge features to CNNs, ViTs, and filtering or target-domain refinement.Surveys identify space-grade deployability, domain-gap reduction, and reliable uncertainty under domain shift as open problems; these methods still assume a known target.Source: §2 · p. 3
• SPE3R is most useful for this work because it covers 64 real satellites with pose labels, but its images are low resolution and are improved here.Other datasets provide different tradeoffs: Dung et al. has real imagery but no metric poses, while URSO, SPEED+ [17], SHIRT, and MAN-DATA provide pose truth for known geometries or one target each.Source: §2 · p. 3
• Unknown-target methods estimate pose or shape but often omit metric scale; several require RGB-D, laser ranging, or stereo, and some report drift or inconsistent tracking.Park et al., Bates et al., DreamSat, DreamSat-Pose, and Augenstein do not estimate scale; Barad et al. report tracking drift, while Dor et al. [24] report inconsistent tracking.Source: §2 · p. 3
• Two monocular unknown-target approaches have limitations: Ticozzi et al. assume perfect initialization and matching and validate only 23 poses, while Legrand et al. require pose estimates for NeRF training.Ticozzi et al. use active SLAM to minimize target shape-model uncertainty; Legrand et al. train a known-target pose network using NeRF-generated imagery.Source: §2 · p. 3
• Terrestrial work includes offline reconstruction and learned feature matching; odometry methods such as FAR use a terrestrial scale prior, while map-free relocalization needs unavailable large training datasets.SuperPoint with LightGlue is one learned feature pipeline, with LightGlue reducing inference time through early stopping.Source: §2 · p. 3
• Accurate monocular pose estimation of unknown spacecraft remains unresolved, motivating this work’s adaptation of terrestrial methods and novel learning-based odometry.Existing space-oriented solutions often rely on additional sensors or do not estimate target range; terrestrial methods do not work out of the box on space-like imagery.Source: §2 · pp. 3–4

3. FEATURE EXTRACTION & MATCHING ON IMAGE PAIRS

• SuperPoint extracts keypoints and descriptors, while LightGlue matches them; poor performance on spacecraft imagery motivates domain adaptation, and target-box crops remove noisy background features.The crops use target bounding boxes derived from reference truth in this work.Source: §3 · p. 4
• LightGlue is fine-tuned on SPE3R+ while SuperPoint remains frozen; reference-truth correspondences use projected features, known poses and metric depth, followed by nearest-neighbor association.Evaluation uses 11 spacecraft withheld during training, representing unknown targets.Source: §3 · p. 4
• SuperPoint’s fixed per-image features let LightGlue matches form multi-image tracks, motivating this pairing over pair-dependent LoFTR correspondences.The resulting tracks supply the MSCKF with geometric constraints described in Section 6.Source: §3 · p. 4

4. ODOMETRY ESTIMATION FROM FEATURE MATCHES

• Both estimators recover only relative rotation and unit translation direction, leaving metric scale to the recovery mechanisms described in Section 5.SON avoids predicting translation scale because it is coupled to target size, which would make generalization to unseen targets more difficult.Source: §4 · p. 4, SON: Spacecraft Odometry Network · p. 4

Classical Approach

• The classical baseline estimates a fundamental matrix from feature matches with OpenCV USAC and MAGSAC++ scoring, then uses known intrinsics and essential-matrix decomposition to recover rotation and translation direction.Source: Classical Approach · p. 4

SON: Spacecraft Odometry Network

• SON regresses relative rotation and unit translation direction from noisy feature matches, using LightGlue transformers and attention-based pooling to provide a fixed-size input to an MLP.Feature locations are mapped to bearing coordinates with the inverse camera matrix, reducing dependence on camera intrinsics, though resolution and blur still affect performance.Source: SON: Spacecraft Odometry Network · pp. 4–5, Attention pooling— · p. 5

Attention pooling—

• Attention pooling reduces the variable number of matched features to a fixed-size representation, and separate MLP heads regress normalized translation direction and rotation.The rotation head maps a continuous 6D representation to SO(3).Source: Attention pooling— · p. 5

Training—

• SON training weights rotation-change bins inversely to their frequency and emphasizes relative accuracy for small-angle pairs, using a 2° floor in the angle normalization.This prevents frequent bins and large-rotation pairs with larger absolute errors from dominating training.Source: Training— · p. 5

SCALE OBSERVABILITY

• SuperPoint-LightGlue features regressed into odometry provide orientation and motion but not target distance, which is needed for collision avoidance.Satellites offer little scale diversity and vary greatly in size; this method reveals distance over time rather than estimating target size, consistent with angles-only navigation when the target appears as a single bright pixel.Source: §5 · p. 5
• Available scale-recovery options are orbit maneuvers, attitude maneuvers, and nonlinear dynamics; the latter is only available at large distances.At the distances considered, relative dynamics are near-linear, so nonlinearities are too weak to rely on.Source: §5 · pp. 5–6
• A boresight roll with the camera’s known nonzero lever arm creates a metric trajectory whose bearing changes make relative range observable.The target is kept at the camera center during the roll. In the mostly along-track separations considered, line-of-sight rotation is only 0.003–0.02°/s—one to two orders of magnitude slower than commanded roll—and too slow to overcome nominal filter drift.Source: §5 · p. 6
• The 0.36 m roll baseline subtends 2.1° at 10 m and 0.7° at 30 m, bounding the range over which the approach remains useful.Source: §5 · p. 6

State

• The nominal state combines relative orbit elements (ROE), target attitude relative to the servicer camera, and angular velocity; attitude clones support measurements spanning multiple images.The filter maintains non-keyframe clones in a sliding window and selected keyframe clones for loop closure; only attitude is cloned.Source: State · p. 6

Dynamics model

• ROE propagate with a J2-only state transition matrix, while attitude propagates kinematically without estimating or assuming known target inertia.The J2-only model is described as appropriate for small spacecraft separations.Source: Dynamics model · p. 7

SON measurement model

• SON rotation updates precede translation-direction updates, which map ROE-derived relative position into the camera frame and normalize the predicted direction.Both sequential updates use numerically linearized measurement models and the Joseph form of the Kalman filter.Source: SON measurement model · pp. 7–8

Feature tracks measurement model

• Feature tracks link SuperPoint observations across images, triangulate temporary feature positions, and remove those positions by null-space projection before updating the filter.A track spanning c images provides 2c−3 constraints; tracks require at least cmin images and are retained only while visible and within the clone window.Source: Feature tracks measurement model · p. 8

Loop closures

• Loop closure retains separated attitude keyframes and uses accepted rotation measurements to remove drift accumulated since a keyframe was last used.Candidates must be at least 2Nw frames old, within 45° predicted relative rotation, have at least 100 LightGlue matches, and pass an NIS threshold of 40.Source: Loop closures · p. 8

ROE fading-memory covariance inflation

• Fading-memory covariance inflation addresses temporal and mutual correlation between SON and feature-track errors that the filter models as independent.It scales the ROE covariance block and cross-covariances while leaving attitude and clone blocks untouched.Source: ROE fading-memory covariance inflation · p. 8

Measurement gating

• A bounded-influence gate limits the effect of outlier SON innovations without discarding measurements; SON channels are skipped when fewer than eight LightGlue matches are available.Feature tracks are separately accepted using a tight gate selected through Monte Carlo trials.Source: Measurement gating · p. 9

Key assumptions

• The evaluation assumes target-pointed camera imagery, exact servicer and camera calibration knowledge, synthetic noise-free images, rigid targets with repeatable visible features, and sufficient onboard compute.The bounding box is derived from a dilated reference truth mask; an upstream method initializes the ROE filter state, and commanded attitude motion has no control error or delay.Source: Key assumptions · p. 9

Datasets

• Four evaluation datasets are used: two terrestrial datasets for LightGlue evaluation, satellite-only SPE3R+ and SHIRT datasets, and rendered trajectories for MSCKF testing.Trajectory poses are generated from reference-truth dynamics, rendered, and used to evaluate the filter.Source: Datasets · p. 9, HPatches— · p. 10

HPatches—

• HPatches evaluates feature matching precision at 3 px reprojection error and homography accuracy using RANSAC and weighted DLT, with AUC reported up to 1 px and 5 px.The dataset provides five image pairs per sequence with known homographies.Source: HPatches— · p. 9
• MegaDepth1500 uses 1,500 image pairs from two scenes to assess feature matching and relative-pose accuracy using essential-matrix estimation within RANSAC.Pose error is computed from maximum angular rotation and translation error, then summarized with AUC at different angular separations.Source: HPatches— · pp. 9–10
• SPE3R+ expands the dataset to 103 spacecraft, with 11 reserved for testing, and renders 5,000 grayscale 1920 × 1200 px images per satellite under varied sunlight.It also provides masks and metric depth for estimating true feature matches used to fine-tune LightGlue.Source: HPatches— · p. 10
• SPE3R+ re-renders the original SPE3R spacecraft imagery at higher resolution and adds 39 models, while rendering spacecraft at varied distances to produce similar angular-size distributions.The original SPE3R dataset contains 64 spacecraft models, with seven reserved for testing, and was rendered at 256 × 256 px; SPE3R+ uses an OpenGL renderer.Source: HPatches— · p. 10
• SON training pairs are rotation-balanced, include large in-plane rotations, and yield 14% of test pairs with total rotation above 30°.LightGlue fine-tuning also includes an equal number of wide-viewpoint pairs for loop closure; 20% of training-set images per satellite are held out for validation.Source: HPatches— · p. 10

• Table 3 summarizes the Monte Carlo campaign and evaluation configuration.Source: Table 3 · p. 11

SHIRT—

• SHIRT contains two Tango rendezvous trajectories, each with 2,371 images at 5 s cadence, in synthetic and real-camera lightbox domains.Tango is not part of SPE3R+ and is therefore an unseen target; SHIRT masks are regenerated for the pipeline.Source: SHIRT— · p. 11
• Because SHIRT has no significant attitude or orbit maneuvers, it provides no range observability and is used only to establish the odometry domain gap.A full-pipeline test with real images is beyond the work’s scope.Source: SHIRT— · p. 11

Reference-truth propagation

• Reference-truth servicer and target orbit and attitude trajectories are propagated jointly using S3, with orbit and dynamics models reported in Table 2.Target inertias are computed from voxelized shapes assuming uniform density and scaled using reported launch masses.Source: Reference-truth propagation · p. 11
• A two-box spacecraft model approximates drag and solar-radiation-pressure torques, with a uniform cuboid fallback; force areas use mean silhouette area across random orientations.The two-box decomposition comprises a central bus and two symmetric thin solar panels; silhouette areas account for panel shading of the bus.Source: Reference-truth propagation · p. 11

Fine-tuning LightGlue

• LightGlue was initially trained on SuperPoint features using homographies sampled from 1 million images, then fine-tuned on the full MegaDepth dataset.This is the original LightGlue training sequence described by its authors.Source: Fine-tuning LightGlue · p. 11
• LightGlue is further fine-tuned on SPE3R+, using the training parameters in Table 1 and the same image splits as SON.LightGlue and SON use different train, validation, and test pair sets, as specified in Table 1.Source: Fine-tuning LightGlue · p. 11

Training SON

• SuperPoint and LightGlue are run on all SPE3R+ image pairs as a preprocessing step to accelerate training; Table 1 lists the baseline SON architecture and settings.Source: Training SON · p. 11

Monte Carlo analysis

• The full navigation architecture is tested on 11 reserved spacecraft, with 50 Monte Carlo trials per target propagated for two orbit periods.Each trial’s relative orbit is scaled to a hand-selected target standoff, balancing improved range observability at closer distances against safety at farther distances.Source: Monte Carlo analysis · p. 11
• Each trial’s ROE are drawn from the Table 3 distribution expressed as fractions of the target’s nominal range ρnom.Source: Monte Carlo analysis · p. 11
• The filter fixes the target-frame attitude gauge by setting initial relative attitude to identity, initializes relative angular velocity to 0, and perturbs initial ROE from reference truth.The ROE perturbation follows the distribution in Table 3.Source: Monte Carlo analysis · p. 11
• Trial metrics use the final 25% of each trajectory; convergence follows Table 3, and specification requires mean attitude error below 15° and total ROE RMSE below 0.1ρ̄.Attitude errors are computed after the constant-rotation alignment described in Section 6; ρ̄ is the mean target–servicer separation.Source: Monte Carlo analysis · p. 11
• Across trials, estimation errors are reported at p50 and p95, while NEES is the median of per-trial evaluation-window means.Source: Monte Carlo analysis · p. 11

Inference analysis

• Inference time and memory are measured per pipeline stage, along with network sizes, on an NVIDIA Jetson AGX Orin 64 GB.Source: Inference analysis · p. 11

MegaDepth1500.

• Fine-tuning LightGlue on SPE3R+ with bounding boxes raises RANSAC AUC by approximately 4–8× across thresholds, but spacecraft-image performance remains worse than on terrestrial images.The improvement accompanies more matches and inliers, while performance on HPatches and MegaDepth1500 decreases; the bounding-box-only case more than quadruples RANSAC AUC but remains insufficient for odometry.Source: MegaDepth1500 · p. 12

Classical odometry estimation

• On SPE3R+, classical translation-direction errors remain around 40–60° across reference rotations, and similar performance across data splits indicates limited sensitivity to spacecraft geometry in fine-tuning.Rotation error increases with reference rotation, while split-wise performance remains too poor for precise pose estimation.Source: Classical odometry estimation · p. 12

SON results

• With nine transformer layers, SON reduces median rotation and translation-direction errors by about 3× and 6× versus one layer; LightGlue initialization adds a small improvement.The MLP-only ablation is on par with or worse than the baseline, indicating transformers are essential among the tested architectures. SON reduces errors versus the classical baseline on held-out spacecraft, though low rotations remain difficult.Source: SON results · pp. 12–13, Table 5 · p. 13

Performance on SHIRT

• With added SHIRT noise, pairs below eight matches increase to 31% for ROE1 and 9% for ROE2; on real lightbox images, SON rotation is on par or sometimes better, while translation-direction error rises 1.5–1.8×.On ROE1, over a quarter of image pairs have reversed translation unit direction; closing the sim-to-real gap through training and noise-model retuning is a prerequisite for flight.Source: Performance on SHIRT · p. 13

Monte Carlo results

• Across the SPE3R+ test satellites on Monte Carlo trajectories, attitude error stays below 12° and ROE error below 10% of total range in more than 95% of cases.Median attitude error is below 6° for every target except SMART-1, which never meets the specification because its dark mylar produces severely underexposed images.Source: Table 7 · p. 14, Monte Carlo results · p. 14
• Removing any tested filter component degrades performance: the classical estimator raises attitude p95 from 11.1° to 93.5°, while removing tracks causes filter failure.Without loop closure, median attitude error rises from 3.9° to 15.6°; without covariance inflation, ROE error grows from 7.4% to 33.8% and NEES from 20.6 to 78.7.Source: Monte Carlo results · pp. 14–15, Table 8 · p. 15

Select Kalman filter results

• For CYGNSS, converged attitude error falls below 2° and angular-velocity error below 0.3°/s; Galileo achieves sub-meter accuracy at about 30 m.The median MSCKF update costs 0.99 s, below the 5 s cadence; feature matching accounts for half of wall-clock time.Source: Select Kalman filter results · pp. 15–16, Table 9 · p. 16

9. CONCLUSION

• SPE3R+ expands the dataset with more, higher-resolution images, metric depth, and more targets; fine-tuned LightGlue and the novel SON enable spacecraft odometry within an MSCKF.The filter tracks ROE, relative attitude, and relative angular velocity.Source: §9 · p. 17
• The MSCKF combines SON measurements with feature-track local constraints and SON-based loop closure for global constraints, while known servicer boresight roll makes target range observable.The geometric constraints address odometry drift, which for attitude is a form of dead reckoning.Source: §9 · p. 17
• Within 12 m, ROE errors are sub-meter in more than 95% of cases and less than 10% of target range; attitude error is below 12° in 95% of cases.These results come from 550 Monte Carlo experiments on 11 spacecraft unseen during training; SMART-1 is reported but excluded from summary statistics because its dark exterior has too little image texture.Source: §9 · pp. 17–18
• On SHIRT hardware-in-the-loop images, SON has 4–6 times lower median translation unit direction error than the classical baseline; the pipeline takes 0.99 s per filter step on an NVIDIA AGX Orin.That runtime is five times faster than the 5 s image cadence; the authors describe this as the first simulation demonstration of monocular metric-scale relative position and attitude estimation for a previously unseen spacecraft.Source: §9 · p. 18
• Future work will address camera-pointing and reference-truth-box assumptions, sim-to-real performance, network size and compute, and inertia estimation for less frequent measurements.The proposed means include a small object detector and estimator-based pointing control; SON training and LightGlue fine-tuning may need changes for real images.Source: §9 · p. 18

BIOGRAPHY

• Pol Francesch Huc is a Space Rendezvous Lab Ph.D. student whose Stanford research focuses on vision-based navigation and characterization of resident space objects.He earned a B.S. in Aeronautical and Astronautical Engineering from Purdue in 2021 and an M.S. in Aeronautics and Astronautics from Stanford in 2024.Source: BIOGRAPHY[ · p. 20
• Simone D’Amico directs Stanford’s Space Rendezvous Laboratory and has 23+ years of experience developing autonomous spacecraft and distributed space systems.He developed and deployed distributed GNC systems for formation-flying, swarming, rendezvous, and proximity-operations missions, and is institutional PI of four NASA- and NSF-funded autonomous satellite swarms.Source: BIOGRAPHY[ · p. 20