PaperlayerView exact revision on arXiv

Source-linked AI summary

Orbit-Planner: Towards Latent World Models for On-Orbit Obstacle Avoidance of Satellite Agents

Zhijian Li, Chao Ren, Peijin Wang, Xian Sun

arXiv:2608.16651v1cs.ROcs.AI

TL;DR

Satellite agents must avoid obstacles under limited observations and dynamic orbital conditions. Orbit-Planner uses action-conditioned latent rollouts with a Physics Probe, achieving a 91.7% success rate in closed-loop Isaac Sim navigation.

  • Problem

    Satellite obstacle-avoidance planners rely on predefined parameters or demonstrations and may generalize poorly when test-time dynamics differ.

  • Method

    Orbit-Planner is a two-stage latent world model that predicts action-conditioned future states and decodes physical-state changes from imagined trajectories.

  • Results

    91.7% success rate was achieved in closed-loop Isaac Sim obstacle avoidance, versus 55.0% for Diffusion Policy across six settings.

  • Takeaways & Limitations

    Orbit-Planner achieves long-horizon latent rollouts, physical-state readouts, and higher closed-loop navigation success than an imitation-learning baseline.

Abstract

from arXiv · show

Satellite agents for on-orbit navigation tasks need to predict collision risks using limited onboard observations. However, conventional planners often rely on predefined maps and fixed environmental assumptions, limiting their adaptability in dynamic on-orbit scenarios. In this paper, we propose Orbit-Planner, a two-stage latent world model for on-orbit obstacle avoidance. Orbit-Planner learns action-conditioned spacecraft dynamics to perform future-state rollouts in latent space, and introduces a Physics Probe to decode physical state changes from imagined latent trajectories. Experiments demonstrate that Orbit-Planner can perform long-horizon latent rollouts and recover physical states from imagined trajectories. In closed-loop obstacle-avoidance navigation in Isaac Sim, it attains a success rate of 91.7%. Code is available at https://github.com/ZhijianLi2003/Orbit_Planner.

I. INTRODUCTION

The introduction motivates adaptive obstacle avoidance for autonomous satellite agents in complex orbital environments and presents Orbit-Planner as a latent world-model solution. The paper contributes an Isaac Sim dataset, a two-stage action-conditioned model with a Physics Probe, and 91.7% online navigation success.

  • Motivation: Autonomous satellite agents require effective obstacle avoidance to support safe and successful missions in complex, dynamic orbital environments.The introduction identifies the growth of space exploration and on-orbit assembly as increasing the importance of autonomous satellite agents.
  • Limitations of Existing Methods: Traditional approaches rely on predefined environmental parameters and classical planners such as Artificial Potential Fields and A* search.
  • Latent World Models: Latent world models offer compact predictive representations that let agents simulate future states and evaluate actions before physical execution.The introduction connects this capability to recent progress in Joint-Embedding Predictive Architectures and dynamic control.
  • Contributions: Orbit-Planner is a two-stage latent world model that learns action-conditioned dynamics for future-state rollouts and uses a Physics Probe to recover physical states from imagined trajectories.
  • Contributions: 91.7% success rate is achieved for online on-orbit obstacle-avoidance navigation based on the proposed world model.

II. METHODOLOGY · A. Problem Formulation

Orbit-Planner formulates CubeSat navigation as a discrete-time control problem in a dynamic, cluttered orbital environment, with the goal of reaching a target while minimizing collision probability. The framework combines action-conditioned latent rollouts with physics-state and obstacle-geometry decoding for trajectory planning.

  • A. Problem Formulation: The CubeSat agent navigates a dynamic, cluttered orbital environment toward a target while minimizing the probability of collision.The system is modeled as a discrete-time dynamical system.
  • A. Problem Formulation: Stage I learns multimodal latent representations and action-conditioned rollouts, while Stage II decodes future state increments and current obstacle geometry for planning.The Physics Probe decodes rolled-out latent trajectories into spacecraft-state increments, and the depth decoder recovers obstacle geometry.
  • A. Problem Formulation: The state includes position displacement pt ∈R3 relative to the initial position, with p0 = 0.Position displacement is expressed in the inertial world frame.
  • A. Problem Formulation: Linear velocity vt ∈R3 and angular velocity ωt ∈R3 are expressed in the spacecraft body frame.These velocity components are part of the CubeSat state representation.
  • A. Problem Formulation: Attitude uses a continuous 6D rotation vector rt ∈R6 derived from the first two columns of Rt ∈SO(3), and fuel is represented by ϕt ∈[0, 1].Rt represents the body-frame attitude in the inertial frame.
  • A. Problem Formulation: The system advances at ∆t = 0.04 s (25 Hz), while at ∈R8 represents normalized thrust commands for the eight onboard thrusters.The control action is defined over the discrete-time dynamics.
  • A. Problem Formulation: The agent assumes its physical state st is fully observable and obtains surrounding environmental information from an onboard RGB camera producing ot ∈RH×W ×3.The camera observations provide visual information about the environment.

B. Orbit-Planner World Model Framework

Orbit-Planner uses a two-stage world-model framework that first performs world-model pre-training and then applies physics probing and planning.

  • Orbit-Planner consists of two stages: world-model pre-training, followed by physics probing and planning.

1) Stage I: World Model Pre-training:

Stage I pre-trains a latent world model through representation and rollout processes to encode spacecraft state and environmental geometry, then predict action-conditioned future latent states. Training combines latent dynamics, variance regularization, and depth supervision to support navigation-relevant representations.

  • Representation: The representation process encodes RGB observations and low-dimensional spacecraft states into a latent representation of the agent and surrounding geometry.A ViT extracts visual tokens, an MLP State Encoder embeds physical state, and a Projector fuses them.
  • Representation: Batch Normalization in the Projector counteracts the ViT’s LayerNorm effects for variance-based regularization.This design helps maintain the variance properties required by SIGReg.
  • Rollout: The rollout process uses an Action Encoder and an autoregressive Transformer with AdaLN to predict future latent states conditioned on thruster commands.The action embedding conditions the AdaLN blocks while the model uses the current latent state and action embedding.
  • Training objectives: Training combines latent dynamics prediction, variance-based regularization, and depth supervision through a dedicated depth decoder.Depth supervision primarily encourages the visual encoder to capture depth cues critical for navigation.

2) Stage II: Physics Probing and Planning:

Stage II freezes the pretrained world model and uses it to predict future latent states from observation, state, and proposed-action sequences. Training uses an observation context length of m = 8 and a rollout horizon of n = 12.

  • 2) Stage II: Physics Probing and Planning:: The frozen world model rolls out future latent states from past observations, corresponding states, and proposed future actions.It predicts z_t+1 through z_t+n for actions a_t through a_t+n−1.
  • 2) Stage II: Physics Probing and Planning:: Training uses an observation context length of m = 8 and a rollout horizon of n = 12.

C. Autonomous Control and Decision-Making

Orbit-Planner uses MPPI to select low-collision-risk action sequences that advance toward the target, executing only the first action before replanning.

  • Autonomous Control and Decision-Making: MPPI uses K = 256 samples over a horizon of H = 50 steps to select the sequence with the lowest predicted collision risk while progressing toward the target.The planner conditions decisions on probe-predicted state rollouts and obstacle point clouds.
  • Autonomous Control and Decision-Making: Only the first action of the selected sequence is executed before replanning.This receding-horizon procedure repeatedly updates control using new predictions.

III. EXPERIMENTS · A. Experimental Setup

Experiments use an Isaac Sim–based space robotics simulation environment with 8,000 on-orbit obstacle-avoidance trajectories. The data are split into training and testing sets, with domain randomization covering obstacle positions and lighting conditions.

  • A. Experimental Setup: The dataset contains 8,000 on-orbit obstacle-avoidance trajectories collected in an Isaac Sim–based space robotics simulation environment.The environment is described as a space robotics simulation platform based on Isaac Sim.
  • A. Experimental Setup: 7,200 trajectories are used for training, while 800 trajectories are reserved for testing.The dataset is split into 7,200 training and 800 testing trajectories.
  • A. Experimental Setup: Domain randomization varies obstacle positions and lighting conditions to improve representation robustness under visual and geometric variation.The randomized conditions are applied during data collection.

B. Latent Space Rollout

Orbit-Planner evaluates action-conditioned multi-step prediction by recursively rolling out future latent states under recorded actions. Although prediction error accumulates with horizon, mean MSE remains moderate at 50 steps, indicating stable long-horizon latent rollouts.

  • Latent Space Rollout: Action-conditioned multi-step prediction recursively rolls out future latents from an encoded context under the recorded action sequence.Errors are measured against encoder-derived target latents at each prediction horizon.
  • Latent Space Rollout: 50-step mean MSE remains moderate despite accumulating autoregressive error, indicating stable long-horizon latent dynamics.Figure 3 reports per-trajectory errors and their mean across rollout horizons.

C. Physics Probe · D. On-Orbit Obstacle Avoidance

The Physics Probe recovers physical-state changes from action-conditioned latent rollouts, while Orbit-Planner uses these imagined futures for closed-loop obstacle avoidance in Isaac Sim. Evaluations cover long-horizon state reconstruction and six randomized obstacle-navigation settings against Diffusion Policy.

  • C. Physics Probe: With the world model’s encoder and predictor frozen, Orbit-Planner encodes an 8-step observation context into initial latent states.The setup evaluates recovery from action-conditioned latent rollouts.
  • C. Physics Probe: The model autoregressively rolls out future latent states conditioned on recorded action sequences.These predicted latents provide the inputs for subsequent physical-state recovery.
  • C. Physics Probe: The Physics Probe maps each predicted latent state to physical-state increments ∆s and integrates them kinematically from the last context state.This reconstructs physical states from imagined trajectories.
  • C. Physics Probe: 50-step (2.0 s) prediction-horizon results compare predicted and ground-truth physical-state components on collision-free trajectories.The qualitative evaluation also compares 3D collision-free and collision windows.
  • D. On-Orbit Obstacle Avoidance: In Isaac Sim, Orbit-Planner uses the frozen world model to imagine action-conditioned futures and select collision-free maneuvers.The closed-loop navigation evaluation tests obstacle avoidance.
  • D. On-Orbit Obstacle Avoidance: Orbit-Planner is compared against the imitation-learning baseline Diffusion Policy under six settings varying obstacle count and initial velocity.The settings cover 1/3/5 obstacles and v0 = 1/2 m/s, with closed-loop success rates and their average reported.
  • D. On-Orbit Obstacle Avoidance: Each navigation setting uses 10 episodes with randomly placed obstacles that are unseen during training.This condition evaluates obstacle avoidance beyond fixed training placements.

IV. CONCLUSION

The paper presents Orbit-Planner, a latent world model that enables autonomous satellite agents to avoid on-orbit obstacles through action-conditioned dynamics and latent future-state rollouts. A Physics Probe decodes physical state transitions from imagined trajectories, supporting long-horizon rollouts, physical-state readouts, and higher closed-loop navigation success than an imitation-learning baseline.

  • Contribution: Orbit-Planner is a latent world model for on-orbit obstacle avoidance by autonomous satellite agents.It is designed for satellite-agent navigation tasks in orbit.
  • Method: Action-conditioned spacecraft dynamics enable future-state rollouts in latent space, while a Physics Probe decodes physical state transitions from imagined trajectories.The two components jointly support prediction and physical-state interpretation of imagined futures.
  • Results: Orbit-Planner achieves long-horizon latent rollouts, physical-state readouts, and higher closed-loop navigation success than an imitation-learning baseline.The conclusion summarizes these experimental capabilities and the comparative navigation result.
  • Future Work: Future work will study the sim-to-real gap under more realistic sensing and dynamics conditions.The proposed evaluation extension targets realism in both sensing and system dynamics.
Loading 2608.16651v1…