Source-linked AI summary
RSL-RL: A Learning Library for Robotics Research
Clemens Schwarke, Mayank Mittal, Nikita Rudin, David Hoeller, Marco Hutter
TL;DR
Robotics researchers need learning libraries that are easier to adapt than broad general-purpose frameworks while supporting robotics-specific algorithms and deployment needs. RSL-RL addresses this with a minimalist, robotics-focused library built around PPO, behavior cloning, auxiliary techniques, and GPU-based training, and it has been used across robotics research and simulator ecosystems. Its scope is intentionally limited: it is not intended for general-purpose machine-learning research, broad RL benchmarking, or native pure imitation learning.
Problem
Broad RL libraries provide algorithmic breadth but can be harder to adapt and extend for robotics, which also requires specialized features such as policy distillation for deployment.
Method
RSL-RL provides a minimalist robotics-focused framework centered on PPO, DAgger-like behavior cloning, symmetry augmentation, curiosity-driven exploration, and GPU-based training.
Results
The library has been used in numerous robotics publications and integrated with NVIDIA Isaac Lab, MuJoCo Playground, and Genesis.
Takeaways & Limitations
RSL-RL supports robotics research aimed at developing learning-based controllers and enhancing real-robot capabilities, including sim-to-real transfer.
Takeaways & Limitations
RSL-RL is not intended for general-purpose machine-learning research, broad RL-algorithm benchmarking, or native pure imitation learning.
Abstract
from arXiv · showhide
RSL-RL is an open-source Reinforcement Learning library tailored to the specific needs of the robotics community. Unlike broad general-purpose frameworks, its design philosophy prioritizes a compact and easily modifiable codebase, allowing researchers to adapt and extend algorithms with minimal overhead. The library focuses on algorithms most widely adopted in robotics, together with auxiliary techniques that address robotics-specific challenges. Optimized for GPU-only training, RSL-RL achieves high-throughput performance in large-scale simulation environments. Its effectiveness has been validated in both simulation benchmarks and in real-world robotic experiments, demonstrating its utility as a lightweight, extensible, and practical framework to develop learning-based robotic controllers. The library is open-sourced at: https://github.com/leggedrobotics/rsl_rl.
1 Introduction
RSL-RL targets robotics researchers who need a compact, adaptable library rather than a broad general-purpose framework. It combines widely adopted robotics algorithms and auxiliary techniques with GPU-based, high-throughput training and packaged simulator integrations.
- General-purpose RL libraries offer broad algorithmic coverage, but their modularity can make robotics code harder to adapt and extend.
- RSL-RL uses a minimalist design centered on widely adopted state-of-the-art algorithms and robotics-specific features.
- Its robotics-first methods include PPO, DAgger-style behavior cloning, symmetry augmentation, and curiosity-driven exploration.
- A GPU-only pipeline supports large-scale batched training with native multi-GPU and multi-node support.
- RSL-RL has been used in robotics publications and integrates with NVIDIA Isaac Lab, MuJoCo Playground, and Genesis for out-of-the-box use.
2 Features
RSL-RL currently provides PPO and DAgger-like behavior cloning, supplemented by robotics-oriented techniques for symmetry, exploration, logging, and evaluation. These features address both policy learning and deployment-related differences between simulation and hardware.
- Algorithms: RSL-RL currently includes PPO and a behavior-cloning algorithm similar to DAgger.PPO is a model-free, on-policy method used for continuous-control tasks including locomotion, manipulation, planning, and navigation.
- Algorithms: Behavior cloning distills an expert policy into a student by iteratively collecting student rollouts, relabeling them with expert actions, and retraining.
- Algorithms: The behavior-cloning workflow is useful after PPO when simulation-only information must be removed for hardware deployment.
- Auxiliary Techniques: Symmetry augmentation mirrors collected states to accelerate sample generation and promote more symmetric behaviors.An additional symmetry loss can reinforce the resulting symmetry.
- Auxiliary Techniques: Curiosity-driven exploration uses a modified Random Network Distillation reward based on a selected subset of the system state.This focuses curiosity on specific parts of the state space in sparse-reward settings.
- Experiment Support: TensorBoard, Weights & Biases, and Neptune provide local or online experiment logging and evaluation options.Cloud services require user accounts and upload data to their platforms.
3 Implementation Details
RSL-RL organizes learning around a Runner, Algorithm, and Network, making most application changes local to those components. Its environment interface requires same-step resets, PyTorch tensors, and TensorDict observations, while PPO includes stabilization details for large-batch training.
- Framework Structure: The framework comprises a Runner, Algorithm, and Network, and users generally modify only these three files.It is implemented in Python and PyTorch for intuitive extensibility.
- Environment Interface: RSL-RL’s VecEnv interface requires environments to implement same-step reset mode.
- Environment Interface: Environment steps must return PyTorch tensors, with observations structured as TensorDicts for selective routing across network modules.TensorDicts also support auxiliary techniques such as RND and latent reconstruction.
- PPO Implementation: Random early termination during initialization mitigates correlated rollouts caused by simultaneous episode termination in large-batch training.The stated effect is improved sample diversity and more stable learning.
4 Applications in Research
RSL-RL originated as a GPU-based, massively parallel simulator framework for legged locomotion and has since supported diverse robotics research applications. These include sim-to-real locomotion, whole-body control, navigation, and teacher-student policy distillation.
- RSL-RL achieved walking policies for legged locomotion in only a few minutes using massively parallel simulation and GPU-based training.
- Researchers have applied the framework to sim-to-real agile locomotion and whole-body control.
- The framework has also supported high-level navigation through mixed action distributions and generalist locomotion through teacher-student distillation.
5 When and When Not to Use RSL-RL
RSL-RL is intended for robotics researchers who need a compact, modifiable learning codebase for developing and validating RL methods, including through sim-to-real transfer. It is not intended as a general-purpose machine-learning or broad RL-benchmarking library.
- RSL-RL targets robotics researchers who need a compact, modifiable, and well-validated learning codebase.
- Its features support performance-oriented research and sim-to-real transfer for validating new robotic learning methods.
- RSL-RL is not intended for fundamental or general-purpose machine-learning research.
- Its limited algorithm set makes straightforward benchmarking of other RL algorithms unsuitable, and it lacks native support for pure imitation learning.