Source-linked AI summary
Reward-Free Continual Adaptation for Resilient Space Robots
Andrej Orsula, Miguel Olivares-Mendez, Carol Martinez
TL;DR
Space robots need continual adaptation despite deployment settings where accurate rewards are often unavailable. The paper introduces a latent-state world-model framework that updates transition dynamics from unsupervised rollouts while freezing the encoder and reward predictor, and evaluates it across three simulated failure scenarios. The reward-free agent initially recovers performance without new rewards, but remains below privileged-reward adaptation and can decay over time.
Problem
Continual RL for degraded space robots requires deployment-time rewards that are often infeasible to compute because external tracking and sufficient sensing are unavailable.
Method
A world model is pre-trained across diverse simulations, then only its transition dynamics are updated from unsupervised rollouts while policy training uses imagined trajectories and the encoder and reward predictor remain frozen.
Results
Across planetary traversal, orbital navigation, and precision assembly, reward-free adaptation initially recovers performance without new rewards but underperforms privileged-reward adaptation and later decays.
Takeaways & Limitations
Latent-state world models can guide continual adaptation when online reward computation is unavailable, though the demonstrated recovery lacks the long-term stability of explicit rewards.
Takeaways & Limitations
The study is simulation-only, and its demanding online optimization exceeds the power constraints of space-grade embedded compute modules.
Abstract
from arXiv · showhide
Space robots operate in extreme environments where hardware degradation can critically compromise traditional control strategies. While continual reinforcement learning offers a promising mechanism for online adaptation, it inherently requires access to a reward signal during deployment. However, precise reward computation in space is often infeasible due to the lack of external tracking systems and the overall complexity of the environment. To address the challenge of unobservable rewards, we introduce a reward-free continual learning framework that leverages latent-state world models. By pre-training a model-based agent across diverse simulations, the world model learns a robust predictor of the reward structure within its latent space. Upon deployment to an environment with severe hardware degradation, we freeze the observation encoder and reward predictor to update only the transition dynamics of the world model through unsupervised rollouts. By training the policy entirely on imagined trajectories generated by this updated world model, the agent adapts to altered dynamics without receiving new rewards. We demonstrate our approach across simulated planetary traversal, orbital navigation, and precision assembly tasks subjected to severe morphological failures.
1 Introduction
Space robots need continual adaptation because hardware degradation can undermine control, yet deployment-time rewards are often unobservable. The proposed framework uses a pre-trained world model’s latent reward landscape to adapt without new rewards across three space-robotics domains.
- Hardware degradation threatens reliable autonomy in long-duration space missions, including failures such as damaged wheels, thrusters, or actuators.
- Continual RL can adapt policies during deployment, but space robots often lack the tracking and sensing needed to compute accurate rewards.
- The framework updates transition dynamics while retaining a pre-trained latent reward structure, enabling policy adaptation without receiving new rewards.
- The study evaluates reward-free adaptation in planetary traversal, orbital navigation, and precision assembly under severe morphological failures.
2 Methodology
The method pre-trains a latent-state world model and policy in diverse simulations, then adapts only transition dynamics from unsupervised rollouts. The encoder and reward-related components remain fixed so imagined trajectories can guide reward-free policy updates.
- 2.1 World Model Pre-Training: The framework builds on DreamerV3, jointly training a latent-state world model and actor-critic policy within imagined rollouts.
- 2.2 Reward-Free Adaptation: During deployment, only the world model’s transition dynamics are updated, while the encoder, decoder, and reward predictor remain frozen.
- 2.1 World Model Pre-Training: The RSSM encodes observations into stochastic latent states and predicts observations, rewards, continuity, and forward dynamics.
- 2.1 World Model Pre-Training: Domain randomization during pre-training is used to make latent representations and reward mappings robust across varied physical conditions.
- 2.2 Reward-Free Adaptation: The zero-shot policy can fail after dynamics shifts because its imagined actions depend on the original transition model.
- 2.2 Reward-Free Adaptation: Online rollouts update transition dynamics, after which the actor-critic is retrained entirely in imagination using the updated model.
- 2.2 Reward-Free Adaptation: The continuity predictor is frozen because termination conditions remain unchanged, but it may require updating when degradation changes episode structure.
3 Experimental Results
Simulation experiments test adaptation to severe failures in three space-robotics tasks under constrained online interaction. Reward-free adaptation recovers performance initially, but remains below privileged-reward adaptation and can decay over time.
- 3.1 Tasks: The experiments cover planetary traversal with wheel failure, orbital navigation with thruster failure, and screwdriving assembly with tool misalignment.
- 3.2 Experimental Protocol: Pre-training uses diverse randomized simulations, whereas online adaptation is restricted to one simulated environment and a 60-minute interaction window.
- 3.3 Performance Evaluation: Four agents are compared: zero-shot, retrained-from-scratch, privileged-reward adaptive, and reward-free adaptive agents.
- 3.3 Performance Evaluation: Zero-shot policies fail catastrophically across all domains after the unmodeled dynamics shift, while retraining confirms the degraded tasks remain solvable.
- 3.3 Performance Evaluation: The privileged-reward agent rapidly approaches the retrained upper bound, indicating fast recovery when true rewards are available.
- 3.3 Performance Evaluation: The reward-free agent shows initial recovery across tasks but consistently underperforms the privileged baseline and exhibits volatility and decay, especially in orbital and assembly tasks.
4 Discussion and Conclusion
The results support using latent-state world models to guide adaptation without observed rewards. However, continual dynamics updates can cause late-stage decay, and the simulation-only, compute-intensive evaluation leaves important deployment boundaries unresolved.
- Experiments validate that diverse-simulation pre-training can encode a latent reward landscape that guides online adaptation without new rewards.
- Late-stage decay reveals an RSSM capacity limit as continued dynamics updates on degraded morphologies overwrite generalized pre-training principles.
- The study is simulation-only and its online optimization exceeds the power constraints of space-grade embedded computing modules.