PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Multi-Task Learning and Online Refinement for Spacecraft Pose Estimation across Domain Gap

Tae Ha Park, Simone D'Amico

arXiv:2203.04275v6cs.CVcs.LG

TL;DR

Synthetic-to-spaceborne visual differences and limited HIL data make spacecraft-pose generalization difficult to evaluate comprehensively. SPNv2 combines multi-task offline training with online entropy-based refinement on unlabeled target images. The combined approach reports 0.15 m and 5.58° errors on lightbox and 0.16 m and 9.79° errors on sunlamp for a larger batch-agnostic variant.

  • Problem

    Synthetic and spaceborne images have fundamentally different visual properties, and available HIL images lack the quantity and diversity to comprehensively evaluate cross-domain generalization.

    Source: §1 · p. 2

  • Method

    SPNv2 jointly trains a shared multi-task encoder on augmented synthetic images, then refines it online on unlabeled target images by minimizing entropy of predicted foreground segmentation.

    Source: §1 · p. 2, §4.1 · p. 7

  • Results

    The combined offline training and ODR yield 0.15 m and 5.58° errors on lightbox and 0.16 m and 9.79° errors on sunlamp for a larger batch-agnostic SPNv2 variant.

    Source: §7 · p. 13

  • Takeaways & Limitations

    The methodology can extend to spaceborne images of a known, noncooperative target collected during future missions.

    Source: §7 · p. 13

    The larger variant is prohibitively large for frequent onboard inference and ODR updates, while the smaller variant averages 0.63 seconds per inference and 1.05 seconds per ODR iteration on an Intel Core i9-9900K CPU.

    Source: §6 · p. 13

Abstract

from arXiv · show

This work presents Spacecraft Pose Network v2 (SPNv2), a Convolutional Neural Network (CNN) for pose estimation of noncooperative spacecraft across domain gap. SPNv2 is a multi-scale, multi-task CNN which consists of a shared multi-scale feature encoder and multiple prediction heads that perform different tasks on a shared feature output. These tasks are all related to detection and pose estimation of a target spacecraft from an image, such as prediction of pre-defined satellite keypoints, direct pose regression, and binary segmentation of the satellite foreground. It is shown that by jointly training on different yet related tasks with extensive data augmentations on synthetic images only, the shared encoder learns features that are common across image domains that have fundamentally different visual characteristics compared to synthetic images. This work also introduces Online Domain Refinement (ODR) which refines the parameters of the normalization layers of SPNv2 on the target domain images online at deployment. Specifically, ODR performs self-supervised entropy minimization of the predicted satellite foreground, thereby improving the CNN's performance on the target domain images without their pose labels and with minimal computational efforts. The GitHub repository for SPNv2 is available at https://github.com/tpark94/spnv2.

1. Introduction

Monocular spacecraft pose estimation matters for autonomous rendezvous, but synthetic-to-spaceborne visual differences and limited HIL data leave generalization difficult to assess. SPNv2 combines related prediction tasks with offline augmentation and online entropy-based refinement; reported results show improved performance on SPEED+ HIL domains.

1. Introduction

  • Monocular pose estimates can support safe, fuel-efficient rendezvous and docking trajectories, using a camera suited to small satellites’ limited onboard capacity.The motivating applications include servicing space assets and active debris removal.Source: §1 · p. 1
  • Synthetic and spaceborne images have fundamentally different visual properties, while SPEED HIL data lack the quantity and diversity needed to comprehensively assess cross-domain generalization.SPEED+ adds lightbox and sunlamp HIL domains to support more comprehensive robustness analysis.Source: §1 · p. 2
  • SPNv2 jointly predicts bounding boxes, pose, heatmaps, and foreground segmentation from a shared encoder, encouraging features for common target geometry and pose across image domains.The introduction attributes this cross-domain feature learning to jointly training related tasks rather than task-specific features tied to synthetic images.Source: §1 · p. 2
  • Offline robust training uses SPEED+ synthetic images with extensive augmentation, while ODR refines the model on unlabeled target images by minimizing segmentation-task Shannon entropy.ODR is presented as source-free and potentially applicable to spaceborne images acquired during missions.Source: §1 · p. 2
  • Offline SPNv2 has translation errors around 22cm/23cm and orientation errors about 8.0°/10.4° on lightbox/sunlamp; ODR further refines these errors after 4096 unlabeled images.For lightbox/sunlamp, ODR reduces translation error by 6cm/7cm and orientation error by 2.4°/0.6° for the larger batch-agnostic variant.Source: §1 · p. 2

2. Related Work

Prior spacecraft pose-estimation work uses CNNs to infer pose through strategies such as attitude classification, regression, and keypoint estimation. The paper situates SPNv2 among these approaches and describes domain-gap evaluation and adaptation challenges in spaceborne vision.

Spacecraft Pose Estimation

  • Earlier spacecraft pose CNNs estimate attitude and translation through classification, regression, keypoints, or heatmaps, with 2D keypoints also supporting full 6D pose recovery through PnP.Examples include probabilistic orientation soft classification and direction-regressed or heatmap-estimated keypoints.Source: §2 · p. 2

Domain Gap in Space

  • Spaceborne domain-gap evaluation remains limited because space access is difficult, leaving prior studies with few real or simulated physical-domain images and limited comprehensive evaluation.Models are commonly trained on abundant synthetic images with augmentation or domain randomization.Source: §2 · p. 3

Source-Free Domain Adaptation

  • Conventional domain adaptation assumes simultaneous source and target data, while source-free approaches avoid that requirement; this work minimizes foreground-prediction entropy and restricts updates to batch-normalization affine parameters.The described design avoids an additional self-supervised proxy task and reduces backpropagation computation.Source: §2 · pp. 3–4

3. Offline Robust Training

Offline robust training combines a multi-scale, multi-task CNN, extensive augmentation, and domain randomization on synthetic data to improve robustness across spacecraft image domains.

3.1. Multi-Scale, Multi-Task Architecture

  • SPNv2 shares an EfficientNet–BiFPN feature encoder across pose, keypoint-heatmap, and foreground-segmentation heads, with the full configuration training all three tasks.The pose head also handles object presence and bounding-box prediction.Source: §3.1 · p. 4

3.2. Normalization Layers

  • GN is used in the prediction heads, while both BN and GN encoders are evaluated for their contribution to resolving the domain gap.The GN heads use 16 channels per group; encoder GN uses group size 8.Source: §3.2 · p. 4

3.3. Training Losses

  • Pose training combines focal classification, complete IoU bounding-box loss, and SPEED pose error, while heatmap and segmentation heads use pixel-wise MSE and binary cross-entropy.All losses are optimized simultaneously with equal weights.Source: §3.3 · pp. 4–5

3.4. Data Augmentation

  • Augmentations are activated with 50% probability, and style transfer randomizes the synthetic spacecraft’s style and inherent texture.Random erase and sun flare are localized to the satellite bounding box.Source: §3.4 · pp. 5–6, Table 1 · p. 5

4. Online Domain Refinement

Online Domain Refinement adapts SPNv2 using unlabeled target-domain images while restricting updates to normalization parameters, supporting sequential and computationally constrained deployment.

4. Online Domain Refinement

  • ODR is designed to be source-free, sequential, and computationally efficient for onboard refinement during rendezvous.The method must avoid waiting for a large sample batch and restrict learning to a small subset of CNN parameters.Source: §4 · p. 6
  • ODR minimizes segmentation-head entropy on target images, updating only the feature encoder’s normalization affine parameters while pose and prediction heads remain fixed.ODR uses the segmentation head for refinement; pose inference uses the heatmap and pose heads.Source: §4 · pp. 6–7, §4.1 · p. 7

4.2. Normalization Layer Parameter Updates

  • For BN encoders, feature mean and variance are updated online every B images using running averages, with images processed sequentially rather than in batches.The running averages combine prior statistics and new statistics over the most recent B images using momentum m.Source: §4.2 · p. 7

5. Experiments

Experiments evaluate offline robust training and online domain refinement (ODR) on SPEED+, with offline training restricted to synthetic data and unlabeled lightbox and sunlamp images used for ODR. They also assess transfer to the limited prisma25 spaceborne set.

5.1. Dataset

  • The evaluation uses 59,960 synthetic images, 6,740 lightbox images, and 2,791 sunlamp images; 25 prisma25 images are reserved for evaluation.Only synthetic training and validation data are used offline; unlabeled lightbox and sunlamp images are used for ODR, while prisma25 is excluded from ODR because its sample count is limited.Source: §5.1 · pp. 7–8

5.2. Metrics

  • Reported measures include bounding-box IoU, rotation and translation errors, and SPEED score; HIL pose errors use calibration-based thresholds, and hH outliers are excluded from reported means.The hH/PnP outliers occur on some HIL images amid domain gap and noisy heatmap predictions.Source: §5.2 · p. 8

5.3. Implementation Details

  • In offline synthetic-domain experiments, SPNv2 uses AdamW and 768 × 512 inputs; training lasts 20 epochs except the hH-only configuration, which lasts 10, with ϕ = 3 and BN by default.ODR receives unaugmented test images one at a time and runs on a single GPU.Source: §5.3 · p. 9

5.4. Results: Offline Robust Training

  • Adding prediction heads improves HIL robustness, while augmentation effects vary by domain: sun flare helps lightbox and style augmentation helps sunlamp.Random erase alone nearly doubles sunlamp pose error; the full configuration performs best among the tested head configurations.Source: Table 2 · p. 8, Table 3 · p. 8, §5.4 · p. 9

5.5. Results: Online Domain Refinement

  • ODR batch size benefits lightbox at B = 4 or 8 but not sunlamp beyond B = 1; longer refinement improves both domains initially, then batch-dependent performance can degrade.For the batch-dependent model, improvement halts after N = 1024 on lightbox and N = 2048 on sunlamp, and lightbox performance falls below baseline after nearly 8000 images; this failure is not observed for the batch-agnostic model.Source: §5.5 · p. 10, Figure 6 · p. 11, Figure 7 · p. 11
  • After ODR, SPNv2 reports translation errors of around 15 cm / 17 cm and orientation errors of 5.6° / 9.7° on lightbox and sunlamp, respectively.SPNv2 with offline training alone also outperforms KRN, including KRN’s lightbox oracle performance; ODR is not run on prisma25 because samples are limited.Source: §5.5 · pp. 10, 13, Table 5 · p. 10

6. Discussion

ODR is computationally efficient in parameter count, but the discussion identifies onboard efficiency and per-image prediction degradation as limitations. The authors call for a smaller batch-agnostic architecture and refinement that avoids significant individual-image losses.

6. Discussion

  • ODR tunes less than 1% of SPNv2’s learnable parameters, but the discussion says further limitations remain before it is fully mission-compliant.Table 6 compares tuned with total learnable parameters in the feature encoder.Source: §6 · p. 13, Table 6 · p. 13
  • The smaller SPNv2 variant takes 0.63 seconds per inference and 1.05 seconds per ODR iteration on an Intel Core i9-9900K, while a bigger variant is prohibitively large for frequent onboard use.The authors seek a smaller, more efficient, batch-agnostic network with cross-domain generalizability at least as good as SPNv2.Source: §6 · p. 13
  • Although ODR improves average HIL performance, its predictions still worsen significantly on many images; future refinement should prevent significant individual-image degradation.The discussion suggests modifying the unsupervised objective or regularizing the loss to achieve non-negative accuracy changes across images.Source: §6 · p. 13

7. Conclusions

SPNv2 combines offline training on augmented synthetic SPEED+ images with online refinement on unlabeled target-domain images. The authors report lightbox and sunlamp pose errors and identify mission-compliance limitations and future evaluation directions.

7. Conclusions

  • SPNv2 is a multi-scale, multi-task CNN trained offline on augmented synthetic SPEED+ images, then refined on unlabeled lightbox and sunlamp images by self-supervised entropy minimization.At deployment, the method refines affine parameters of the feature extractor.Source: §7 · p. 13
  • The larger, batch-agnostic SPNv2 variant reports 0.15 m and 5.58◦ errors on lightbox, and 0.16 m and 9.79◦ errors on sunlamp.The conclusion attributes these results to the combination of offline training and ODR.Source: §7 · p. 13
  • Mission-compliant use requires a model that is fully batch-agnostic, less computationally demanding, and avoids degrading predictions on target samples during ODR.Future work will integrate SPNv2 and ODR into a navigation filter and evaluate prediction uncertainty and its impact on filter performance.Source: §7 · p. 13
Loading 2203.04275v6…