A framework for Reinforcement Learning research.

│ Overview │ Getting Started │ Documentation │ Citation │

Overview

Highlights

• 💡 Perfect to understand and prototype algorithms:
  • One algorithm = One directory -> No backtracking through parent classes
  • Algorithms can be easily copied out of RL-X
• ⚒️ Known DL libraries: Implementations in PyTorch and JAX
• ⚡ Maximum speed: Just-In-Time (JIT) compilation and parallel environments
• 🧪 Mix and match and extend: Generic interfaces between algorithms and environments
• ⛰️​ Custom environments: Examples for MuJoCo, Isaac Lab, ManiSkill or custom socket communication
• 🚀​ GPU environments: MJX, Warp, Isaac Lab and ManiSkill can run thousands of parallel environments
• 🤖​ Robot learning: Training and deployment for locomotion and motion tracking with the G1 and Go2
• ⚽ RoboCup: Training for the RoboCup soccer competition in MuJoCo and MJX
• 🕰️ Memory architectures: PPO with GRU, LSTM, Transformer, History Window, Mamba-2, Memory Actions
• 📈 Experiments: Checkpoints, Evaluation, Console log, Tensorboard, Weights & Biases, SLURM, Docker

Implemented Algorithms

• Proximal Policy Optimization (PPO)
• Proximal Policy Optimization + Differentiable Trust Region Layers (PPO+DTRL)
• Proximal Policy Optimization + Gated Recurrent Unit (PPO+GRU)
• Proximal Policy Optimization + Long Short-Term Memory (PPO+LSTM)
• Proximal Policy Optimization + Transformer (PPO+Transformer)
• Proximal Policy Optimization + History Window (PPO+HistoryWindow)
• Proximal Policy Optimization + Mamba-2 (PPO+Mamba-2)
• Proximal Policy Optimization + Memory Actions (PPO+MemoryActions)
• Early Stopping Policy Optimization (ESPO)
• Trust Region Policy Optimization (TRPO)
• Simple Policy Optimization (SPO)
• Flow Policy Optimization (FPO)
• Diffusion Policy Policy Optimization (DPPO)
• Diffusion Models for Maximum Entropy Reinforcement Learning (DIME)
• Relative Entropy Pathwise Policy Optimization (REPPO)
• Deep Deterministic Policy Gradient (DDPG)
• Twin Delayed Deep Deterministic Gradient (TD3)
• Fast Twin Delayed Deep Deterministic Gradient (FastTD3)
• Soft Actor Critic (SAC)
• Fast Soft Actor Critic (FastSAC)
• Flash Soft Actor Critic (FlashSAC)
• Randomized Ensembled Double Q-Learning (REDQ)
• Dropout Q-Functions (DroQ)
• Bigger, Regularized, Optimistic (BRO)
• CrossQ
• XQC
• Simplicity Bias (SimBa)
• Simplicity Bias V2 (SimBaV2)
• Truncated Quantile Critics (TQC)
• Aggressive Q-Learning with Ensembles (AQE)
• Maximum a Posteriori Policy Optimization (MPO)
• Fast Maximum a Posteriori Policy Optimization (FastMPO)
• Deep Q-Network (DQN)
• Deep Q-Network with Histogram Loss using Gaussians (DQN HL-Gauss)
• Double Deep Q-Network (DDQN)
• Categorical Deep Q-Network (C51)
• Parallelized Q-Network (PQN)

Usable Environments

• Gymnasium
  • MuJoCo
  • Atari
  • Classic control
  • DeepMind Control Suite
• EnvPool
  • MuJoCo
  • Atari
  • Classic control
  • DeepMind Control Suite
• MuJoCo Playground
  • Locomotion
• Custom MuJoCo
  • Ant velocity-tracking: Examples with MuJoCo, MJX, MJX Warp and Warp Torch backends
  • Robot locomotion: Go2/G1 locomotion training and real-robot deployment
  • Robot motion tracking: BeyondMimic-style G1 tracking of retargeted LAFAN/OMOMO motions
  • Custom RoboCup Soccer: Example of custom MuJoCo and MJX environments for the RoboCup soccer simulation 3D league and other humanoid soccer leagues
• Custom Isaac Lab
  • Example of a custom Isaac Lab environment
• Custom ManiSkill
  • Example of a custom ManiSkill environment
• Custom Interface
  • Prototype of a custom environment interface with socket communication

All listed environments are directly embedded in RL-X and can be used out-of-the-box.

For further information on the environments (README) and algorithms (README) and how to add your own, read the respective README files.

Getting Started

Install

Default installation for a Linux system with a NVIDIA GPU:

conda create -n rlx python=3.11.4
conda activate rlx
git clone git@github.com:nico-bohlinger/RL-X.git
cd RL-X
pip install -e .[all] --config-settings editable_mode=compat
pip uninstall $(pip freeze | grep -E '\-cu12|\-cu13' | cut -d '=' -f 1) -y
pip install "torch>=2.7.0" --index-url https://download.pytorch.org/whl/cu118 --upgrade
pip install "jax[cuda12]"

For other configurations, see the detailed installation guide in the documentation. As Isaac Lab needs to be installed separately, instructions can also be found there. Similarly, ManiSkill might need additional steps, like downgrading numpy.

Example

cd experiments
python experiment.py

Detailed instructions for running experiments can be found in the README file in the experiments directory or in the documentation.

Example for Google Colab:

Citation

If you use RL-X in your research, please cite the following paper:

@incollection{bohlinger2023rlx,
      title={RL-X: A Deep Reinforcement Learning Library (not only) for RoboCup}, 
      author={Nico Bohlinger and Klaus Dorer},
      booktitle={Robot World Cup},
      pages={228--239},
      year={2023},
      publisher={Springer}
}