PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning in Orbit: Onboard Training and Closed-Loop Attitude Control with a Thermal Segmentation Model on a Spacecraft

Elena Sorina Lupu, Hannah Grauer, Phaedrus Leeds, Nikhil Ranganathan, Connor Lee, Sulekha Kishore, Cailyn Smith, Shon Cortes, Brian Hardy, Benjamen Bycroft, Darren W. Rowen, Soon-Jo Chung

arXiv:2610.19122v1cs.ROcs.AIcs.SYeess.SY

TL;DR

Deep-space missions need spacecraft autonomy where communication delays and bandwidth limits make ground operation impractical. This work trains and runs a thermal segmentation model onboard a 6U CubeSat and uses it for closed-loop attitude control toward the Moon. Across 12 onboard experiments, the authors report sufficient performance for closed-loop control under LEO conditions.

  • Problem

    Deep-space missions need spacecraft that can perceive, learn, and act autonomously because communication delays, limited bandwidth, and mission complexity make reliance on ground operation impractical.

    Source: §I · p. 1

  • Method

    The authors demonstrate onboard training and inference of a thermal-image segmentation model on a 6U CubeSat, integrating its output with the attitude control system to point toward the Moon.

    Source: §VII · p. 13

  • Results

    Across 12 onboard training and inference experiments on an NVIDIA Jetson TX2 NX GPU in LEO, the perception model achieved performance sufficient for closed-loop control.

    Source: §VII · p. 13

  • Takeaways & Limitations

    The mission’s experiments show both the promise and challenges of deploying learning-based autonomy in space.

    Source: §VII · p. 13

    Supervised learning relies on labeled data, which the authors say is restrictive for more complex space tasks and impractical to acquire in many applications.

    Source: A. Algorithm Performance · p. 12

Abstract

from arXiv · show

Deep space missions increasingly require spacecraft capable of autonomously operating in environments with communication delays and bandwidth limitations. A key enabler of such autonomy is the ability to run learning-based algorithms directly onboard. In this work, we present a flight deployment on a 6U CubeSat that demonstrates (1) onboard training and inference of a thermal segmentation model on an NVIDIA Jetson TX2 NX GPU in space and (2) closed-loop attitude control using the segmentation model on a real-time, thermal image feed to track the Moon. In addition to this demonstration, we implement ROS 2 Foxy as a data transport middleware and integrate an open-source file system that provides strong data integrity guarantees and self-healing features. Over the course of approximately one year, we performed 12 experiments on the spacecraft, advancing the state of the art in autonomous spacecraft systems by showing how modern learning-based pipelines, commercially available hardware, and open-source software can operate reliably under the compute and environmental constraints of Low Earth Orbit (LEO).

I. INTRODUCTION

Spacecraft autonomy matters for missions where communication delays, limited bandwidth, and mission complexity make reliance on ground operation impractical. This work demonstrates onboard learning and closed-loop control, alongside a review of flown learning demonstrations.

I. INTRODUCTION

  • Communication delays, limited bandwidth, and mission complexity make reliance on ground operation impractical for deep-space autonomy.Autonomous spacecraft need to perceive, learn, and act without a human in the loop.Source: §I · p. 1

A. Contributions

  • The flight demonstration trains a vision segmentation model onboard and uses it to command spacecraft attitude toward the Moon.The authors report training and inference on a CubeSat’s NVIDIA Jetson TX2 NX GPU.Source: A. Contributions · p. 2
  • The contributions also include ROS 2 Foxy between the segmentation module and bus avionics, OpenZFS, and CPU/GPU and thermal measurements.The authors describe OpenZFS as providing data-corruption and repair guarantees and robustness to sudden power loss.Source: A. Contributions · p. 2

A. Flown Missions with Learning Demonstrations

  • Earlier flown demonstrations included Deep Space 1’s symbolic reasoning and planning, EO-1’s learning methods, and DEMETER’s neural network for scientific signal processing.The cited examples also include IPEX, whose random forest achieved high accuracy after prelaunch training on four hand-labeled balloon-test images.Source: A. Flown Missions with Learning… · p. 3
  • OPS-SAT supported CNN inference and live-data retraining, while Phi-Sat-1 validated real-time cloud detection using an Intel Movidius Myriad 2 VPU.The passages also describe commercial platforms extending onboard training and inference, including ION Satellite Carrier and CogniSAT-6.Source: A. Flown Missions with Learning… · p. 3

B. Closed-loop Control using Vision

Vision-based control integrates perception outputs into feedback loops in terrestrial robotics, while space uses of vision have largely centered on navigation and Earth observation. ROS 2 offers middleware capabilities, but had not been demonstrated on a free-flying spacecraft in orbit.

B. Closed-loop Control using Vision

  • Terrestrial vision-based control integrates perception outputs into feedback loops, from visual servoing to learned control of robots.The cited work spans simulated agents, ground robots, and robot manipulation.Source: B. Closed-loop Control using Vision · p. 3
  • In space, vision has primarily supported navigation, pose estimation, rendezvous and proximity operations, and Earth-observation processing.Examples include thermal and LiDAR relative navigation for Dragon docking and image-informed guidance for planetary landing and hazard detection.Source: B. Closed-loop Control using Vision · p. 3

C. ROS 2 Middleware

  • ROS 2 extends ROS middleware with improved real-time performance and distributed communication via DDS.ROS provides hardware abstraction and a communication framework used widely in terrestrial robotics.Source: C. ROS 2 Middleware · p. 3
  • Despite efforts to adapt ROS 2 for space, it had not been demonstrated on a free-flying spacecraft in orbit.The passage identifies a gap in reusable, accessible flight-software frameworks for robotic space systems.Source: C. ROS 2 Middleware · p. 3

III. THE EDGE NODE MISSION

Edge Node Lite is a precursor mission designed to reduce risk by validating onboard image-processing and machine-learning technologies in orbit. Its payload combines commercial computing and sensing hardware with spacecraft interfaces.

A. Edge Node Lite

  • Edge Node Lite was designed to reduce risk for the full Edge Node mission by demonstrating advanced onboard image processing and machine learning.The payload includes a subset of sensing and onboard computing hardware for in-orbit validation.Source: A. Edge Node Lite · p. 4
  • Edge Node Lite demonstrated onboard ML training, ROS 2 deployment in orbit, and containerized software using Docker.The mission also aimed to accelerate adoption of compact commercial computing and sensing technologies.Source: A. Edge Node Lite · p. 4
  • The payload pairs an NVIDIA Jetson TX2 NX with 1 TB NVMe storage, a radiation-tolerant ARM Cortex-M7, and spacecraft data and bus interfaces.The Jetson runs Ubuntu 18.04 and uses ROS 2 Foxy as middleware.Source: A. Edge Node Lite · p. 4
  • A FLIR Boson+ long-wave infrared camera supplies the segmentation images and was chosen for day/night invariance.Its uncooled VOx microbolometer operates in the 8–14 μm wavelength range.Source: A. Edge Node Lite · pp. 4–5

B. 6U CubeSat

  • The 6U CubeSat operates in a sun-synchronous, near-polar orbit at approximately 500 km altitude, with an orbital period of about 94 minutes.The spacecraft has no propulsion, is eclipsed for about 30% of its orbit, and thermally sinks the Jetson to the spacecraft wall.Source: B. 6U CubeSat · p. 5
  • The customized operating system includes ROS 2-published resource and temperature monitoring, camera collection and telemetry tools, and ML packages.Disk-backed swap and boot-flow changes support virtual memory and the ZFS filesystem; the spacecraft also supported a lake-and-island identification demonstration.Source: B. 6U CubeSat · p. 5

A. Problem Statement

The paper demonstrates vision-based spacecraft attitude control in LEO using an onboard-trained segmentation model, with the Moon as the target. It describes the model’s distillation and in-orbit training, then uses segmented images to support Moon-pointing commands.

A. Problem Statement

  • The system uses an onboard-trained segmentation model to estimate the Moon’s image position and generate attitude commands for spacecraft pointing.The work is a technology demonstration; other vision-based spacecraft-control approaches remain possible.Source: A. Problem Statement · p. 5

1) Segmentation Module

  • Fast-SCNN is distilled from DINO using ImageNet-21k data, then fine-tuned on in-orbit images for segmentation.The distillation matches student-network features to DINO’s intermediate features; the segmentation head is not involved in that loss.Source: 1) Segmentation Module · p. 5
  • Onboard, the DINO-distilled Fast-SCNN performs inference and fine-tuning on thermal images using the spacecraft’s Jetson TX2 NX GPU.The spacecraft’s thermal images have one channel, rather than the three channels used for RGB images.Source: 1) Segmentation Module · pp. 5–6
  • Dynamic targeting extracts image-plane centroids, filters their coordinates with a Kalman Filter, and sends them to the ADCS for Moon pointing.The filter assumes constant-velocity target motion in the image plane, which the authors consider reasonable for low slew rates.Source: 1) Segmentation Module · p. 6

V. RESULTS

The results section covers 12 onboard experiments and ground testing, reporting segmentation performance alongside computing-resource and thermal measurements.

V. RESULTS

  • 12 onboard spacecraft experiments and additional ground-testing results are presented.The onboard experiments include inference and training; the paper also uses ground testing to validate its solution.Source: §V · p. 7
  • mIoU is the performance metric reported for most experiments, averaging classwise intersection-over-union scores.An mIoU of 1.0 indicates a perfect prediction, while 0.0 indicates a complete mismatch.Source: §V · p. 7
  • CPU/GPU usage and thermal performance are also reported during training and inference.Source: §V · p. 7

A. Training Results on Spacecraft

The spacecraft trained a DINO-distilled Fast-SCNN model onboard using ground-generated labels and a representative set of 40 Moon images. Training reached a final mIoU above 0.9, and the resulting weights were stored for later inference and dynamic targeting.

A. Training Results on Spacecraft

  • Final mIoU exceeded 0.9 after 60 epochs of onboard training, run in two 30-epoch phases lasting approximately 13 minutes total.Performance improved rapidly after the first epoch and then stabilized.Source: A. Training Results on Spacecraft · p. 7, Figure 4 · p. 7
  • Training used uploaded ground-generated labels and 40 selected Moon images spanning different phases and overexposed cases.The model was initialized from weights loaded before launch.Source: A. Training Results on Spacecraft · p. 7
  • The final model weights were stored onboard for subsequent inference experiments and dynamic targeting.The two-phase procedure also demonstrated fine-tuning from weights generated on orbit.Source: A. Training Results on Spacecraft · p. 7
  • During the eclipse experiment, GPU temperature rose by about 1–4°C during training, and GPU usage was more pronounced during training than inference.The overall temperature trend decreased as the spacecraft spent more time in eclipse; the training runs were separated by a Docker-container restart.Source: Figure 4 · p. 7, A. Training Results on Spacecraft · p. 8

B. Dynamic Targeting

The onboard-trained segmentation model fed a Kalman-filtered Moon position into the spacecraft attitude-control system for closed-loop targeting. Tracking oscillations were reduced in later runs, while the year-in-orbit checks reported no ZFS checksum errors and no observed signs of radiation-induced effects.

B. Dynamic Targeting

  • The onboard segmentation model detected the Moon in real time; filtered image-plane coordinates drove attitude commands to reorient the spacecraft toward it.Inference events produced bursty GPU utilization, with bursts around 3–4 seconds in the 1 Hz recording.Source: B. Dynamic Targeting · p. 8
  • Later tracking experiments reduced the horizontal oscillation seen in early runs after changes to frame buffering, inference frequency, control gain, and filtering.Inference and Kalman-filter prediction rates were raised from 1 Hz to 2 Hz; a low-pass filter was later disabled as counterproductive for fast tracking with relatively low measurement noise.Source: B. Dynamic Targeting · p. 8
  • Dynamic vision-based closed-loop control was achieved onboard under realistic compute constraints, with Run 7 showing the Moon moving toward image center.Figure 5 includes GPU temperature, four-core CPU usage, GPU load, segmented and thermal image sequences, and run command traces.Source: B. Dynamic Targeting · p. 8, Figure 5 · p. 9

C. Performance of the NVIDIA Jetson TX2 NX in LEO

D. Radiation Testing

The radiation tests assessed NVMe SSD behavior under proton exposure, while implementation work integrated ROS 2 middleware and OpenZFS into the spacecraft software stack. The filesystem configuration used duplication for critical operating-system files and a launch-time snapshot for recovery.

1) Solid State Drive (SSD)

  • Seven SSDs across four models underwent about eight hours of cumulative 202 MeV proton exposure during continuous write-read-verify testing.The tests tracked throughput and data integrity and detected stalled I/O or major throughput degradation.Source: 1) Solid State Drive (SSD) · pp. 9–10
  • Western Digital devices had 10 times fewer SEFI faults than Greenliant or Swissbit drives, and one Greenliant device was destroyed.For an approximately 500 km orbit, the passage estimates about 0.5 SEFI per year and a 10% probability of permanent failure after two years; it recommends the WD IX SN530 1TB for LEO.Source: 1) Solid State Drive (SSD) · p. 10
  • The SSD tests accessed drives as block devices, so filesystem behavior was not part of the radiation-test results.The tests preceded OpenZFS support.Source: 1) Solid State Drive (SSD) · p. 10

E. Implementation Details

  • ROS 2 Foxy with Fast DDS used shared memory for process communication, while a micro-ROS Agent bridged the Jetson network to the supervisor microcontroller.The microcontroller interfaced with avionics telemetry and forwarded Jetson attitude commands to the attitude-control board.Source: E. Implementation Details · p. 10

2) File System (ZFS)

  • OpenZFS was integrated with the Linux for Tegra stack and U-Boot, requiring patches that were upstreamed to U-Boot.Source: 2) File System (ZFS) · p. 10
  • ZFS uses copy-on-write and checksums to support recovery from sudden power loss and detection or repair of data corruption; critical operating-system files were duplicated, while sensor data were not.A launch-time filesystem snapshot could be restored through a microcontroller-controlled GPIO pin; RAID-Z was not used.Source: 2) File System (ZFS) · p. 10

3) Codebase

The codebase used Python and Bash, and ground validation drew on spacecraft imagery and thermal datasets. The RGB evaluation and model-distillation ablation used established train/test/validation splits, while inference timing was benchmarked on the flight computer and a workstation GPU.

3) Codebase

  • The segmentation experiment was implemented in Python 3, with experiment steps orchestrated by Bash scripts.Source: 3) Codebase · p. 10

1) Datasets

  • Ground validation included simulator images of spacecraft parts, a public 3,117-image satellite and space-station dataset, synthetic thermal imagery, and thermal Moon images captured on the spacecraft.Landsat 8/9 imagery supplied Earth backgrounds varying in location, cloud coverage, and solar angle.Source: 1) Datasets · p. 11, Figure 6 · p. 11

2) Performance on the RGB datasets on the ground

3) Model Distillation Ablation Study

  • Both Fast-SCNN and DINO-distilled Fast-SCNN performed better with larger training datasets in the custom spacecraft-parts and Adelaide ablations.The study compared training-set fractions of 75%, 50%, and 12.5%, with 5% validation and 20% testing data.Source: 3) Model Distillation Ablation Study · p. 11

4) Compute results

  • Inference time for short-range detection models was benchmarked on the Jetson TX2 NX and Titan RTX.The Jetson TX2 NX was used because it matched the computer on the spacecraft; the Titan RTX provided a workstation-GPU comparison.Source: 4) Compute results · p. 11

VI. DISCUSSIONS

The discussion reports strong on-orbit segmentation and dynamic-targeting performance while identifying image variability, supervision dependence, and power behavior as practical constraints. It also describes a training-related brown-out during testing and a change to the Jetson’s power mode.

A. Algorithm Performance

  • On-orbit segmentation and dynamic targeting performed strongly despite streaking, overexposure, and changing illumination that complicated consistent segmentation.These image effects introduced noise and variability.Source: A. Algorithm Performance · p. 11
  • Supervised learning remains restrictive for complex space tasks because labels may be impractical to acquire, although uploading labels required 172 KB versus 6.5 MB for weights.The authors identify autonomous adaptation without ground-provided supervision as an ideal for onboard systems.Source: A. Algorithm Performance · p. 12
  • Imperfect Moon boundaries varied the estimated centroid, adding measurement noise that appeared as small spacecraft-pointing oscillations.Improved segmentation, out-of-distribution detection, and Kalman-filter tuning are expected to reduce these effects and enhance stability.Source: A. Algorithm Performance · p. 12

1) Brown-out during inference and training

  • After about 30 seconds of training on an engineering model, the Jetson TX2 browned out when current spikes caused voltage to fall below the required 12 V.The power mode was changed from 20 W to 15 W; the modes affect CPU, GPU, and RAM clock speeds.Source: 1) Brown-out during inference and… · p. 12
  • An infinite-loop matrix multiplication used for earlier power testing had higher average power consumption than the flight workload but was assumed to represent GPU neural-network training.The supplied discussion also identifies an inference-time table for two platforms, but provides no values to compare.Source: 1) Brown-out during inference and… · p. 12, Table 8 · p. 12

2) ROS 2 Bag Compression

The authors disabled ROS 2 bag compression during recording because finalization delays and a possible uncommitted database state posed operational concerns. The spacecraft architecture also layers safeguards to limit payload operation and support recovery from faults.

2) ROS 2 Bag Compression

  • For a representative 5 GB ROS 2 bag, compression to about 4 GB delayed finalization by roughly 10 minutes, so compression was moved to downlink preparation.The authors also cite the risk of a database remaining in an uncommitted Write-ahead Log state, making recovery difficult.Source: 2) ROS 2 Bag Compression · p. 12

1) Payload-Level Safeguards •

  • Payload power is Flight Computer-controlled and typically limited to under an hour; its microcontroller can still gather basic telemetry if the Jetson fails to boot.Experiments run in Docker containers with limited filesystem access, while the bus shell-command service has two dedicated CPU cores.Source: 1) Payload-Level Safeguards • · pp. 12–13

2) Flight Computer Watchdog •

  • A hardware watchdog resets the Flight Computer if its lowest-priority idle task fails to service it at least once every 10 seconds.The reset detects conditions in which higher-priority tasks stall and prevent the idle task from running.Source: 2) Flight Computer Watchdog • · p. 13

3) Software Resilience •

  • A read-only golden image boots on every reset, while a failed patchable software image triggers automatic reversion to the golden image on the next boot.Source: 3) Software Resilience • · p. 13

4) Power Management and Safe Modes •

  • Battery-voltage and temperature thresholds can trigger safe mode, which powers off payloads and idles avionics; main power reset can also cycle most avionics to recover from faults.Source: 4) Power Management and Safe Modes • · p. 13

5) Communication Resilience •

The paper describes communication resilience, camera streaks, and operational lessons alongside a 6U CubeSat flight demonstration of onboard learning and closed-loop autonomy. Across 12 LEO experiments, the onboard-trained model provided sufficient performance for closed-loop control.

5) Communication Resilience •

D. Camera Streaks

  • FLIR Boson+ images in space showed streak patterns believed to arise when the Sun passed through the frame during slews; increasing Flat Field Correction cadence modestly reduced them.Flat Field Correction captures a reference frame with the shutter briefly closed, then subtracts that pattern from subsequent images to address fixed-pattern noise and sensor variation.Source: D. Camera Streaks · p. 13

E. Operations

  • Containerized components and Docker Compose supported modular deployment and updates, while ROS 2 bags and standard tooling enabled playback and analysis.The workflow required uploading lightweight labels rather than new model weights.Source: E. Operations · p. 13

VII. CONCLUSIONS

  • Twelve LEO experiments demonstrated onboard training and inference on a Jetson TX2 NX GPU, with performance sufficient for closed-loop control.The model was distilled from a vision foundation model and achieved good segmentation performance on a limited set of Moon images captured in orbit.Source: §VII · p. 13
  • The risk-tolerant autonomy stack uses ROS 2 with Fast DDS and Micro-XRCE-DDS for data transport and Docker-based containerization for deployment.Source: §VII · p. 13
  • The authors report thermal telemetry and CPU/GPU usage during training and inference, and discuss ground-testing challenges as lessons on the promise and challenges of learning-based autonomy in space.Source: §VII · p. 13
Loading 2610.19122v1…