Installation
Prerequisites
skrl requires Python 3.6 or higher and the following libraries (they will be installed automatically):
Library Installation
GitHub repository
Clone or download the library from its GitHub repository (https://github.com/Toni-SM/skrl)
git clone https://github.com/Toni-SM/skrl.git cd skrl
Install in editable/development mode (links the package to its original location allowing any modifications to be reflected directly in its Python environment)
pip install -e .
Install in the current Python site-packages directory (modifications to the code downloaded from GitHub will not be reflected in your Python environment)
pip install .
Troubleshooting
To ask questions or discuss about the library visit skrl’s GitHub discussions
https://github.com/Toni-SM/skrl/discussions
Bug detection and/or correction, feature requests and everything else are more than welcome. Come on, open a new issue!
Known issues
When using the parallel trainer with PyTorch 1.12
See PyTorch issue #80831
AttributeError: 'Adam' object has no attribute '_warned_capturable_if_run_uncaptured'
Changelog
# Changelog
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
## [0.10.2] - 2023-03-23
### Changed
- Update loader and utils for OmniIsaacGymEnvs 2022.2.1.0
- Update Omniverse Isaac Gym real-world examples
## [0.10.1] - 2023-01-26
### Fixed
- Tensorboard writer instantiation when `write_interval` is zero
## [0.10.0] - 2023-01-22
### Added
- Isaac Orbit environment loader
- Wrap an Isaac Orbit environment
- Gaussian-Deterministic shared model instantiator
## [0.9.1] - 2023-01-17
### Added
- Utility for downloading models from Hugging Face Hub
### Fixed
- Initialization of agent components if they have not been defined
- Manual trainer `train`/`eval` method default arguments
## [0.9.0] - 2023-01-13
### Added
- Support for Farama Gymnasium interface
- Wrapper for robosuite environments
- Weights & Biases integration (by @juhannc)
- Set the running mode (training or evaluation) of the agents
- Allow clipping the gradient norm for DDPG, TD3 and SAC agents
- Initialize model biases
- Add RNN (RNN, LSTM, GRU and any other variant) support for A2C, DDPG, PPO, SAC, TD3 and TRPO agents
- Allow disabling training/evaluation progressbar
- Farama Shimmy and robosuite examples
- KUKA LBR iiwa real-world example
### Changed
- Forward model inputs as a Python dictionary [**breaking change**]
- Returns a Python dictionary with extra output values in model calls [**breaking change**]
- Adopt the implementation of `terminated` and `truncated` over `done` for all environments
### Fixed
- Omniverse Isaac Gym simulation speed for the Franka Emika real-world example
- Call agents' method `record_transition` instead of parent method
to allow storing samples in memories during evaluation
- Move TRPO policy optimization out of the value optimization loop
- Access to the categorical model distribution
- Call reset only once for Gym/Gymnasium vectorized environments
### Removed
- Deprecated method `start` in trainers
## [0.8.0] - 2022-10-03
### Added
- AMP agent for physics-based character animation
- Manual trainer
- Gaussian model mixin
- Support for creating shared models
- Parameter `role` to model methods
- Wrapper compatibility with the new OpenAI Gym environment API (by @juhannc)
- Internal library colored logger
- Migrate checkpoints/models from other RL libraries to skrl models/agents
- Configuration parameter `store_separately` to agent configuration dict
- Save/load agent modules (models, optimizers, preprocessors)
- Set random seed and configure deterministic behavior for reproducibility
- Benchmark results for Isaac Gym and Omniverse Isaac Gym on the GitHub discussion page
- Franka Emika real-world example
### Changed
- Models implementation as Python mixin [**breaking change**]
- Multivariate Gaussian model (`GaussianModel` until 0.7.0) to `MultivariateGaussianMixin`
- Trainer's `cfg` parameter position and default values
- Show training/evaluation display progress using `tqdm` (by @juhannc)
- Update Isaac Gym and Omniverse Isaac Gym examples
### Fixed
- Missing recursive arguments during model weights initialization
- Tensor dimension when computing preprocessor parallel variance
- Models' clip tensors dtype to `float32`
### Removed
- Parameter `inference` from model methods
- Configuration parameter `checkpoint_policy_only` from agent configuration dict
## [0.7.0] - 2022-07-11
### Added
- A2C agent
- Isaac Gym (preview 4) environment loader
- Wrap an Isaac Gym (preview 4) environment
- Support for OpenAI Gym vectorized environments
- Running standard scaler for input preprocessing
- Installation from PyPI (`pip install skrl`)
## [0.6.0] - 2022-06-09
### Added
- Omniverse Isaac Gym environment loader
- Wrap an Omniverse Isaac Gym environment
- Save best models during training
## [0.5.0] - 2022-05-18
### Added
- TRPO agent
- DeepMind environment wrapper
- KL Adaptive learning rate scheduler
- Handle `gym.spaces.Dict` observation spaces (OpenAI Gym and DeepMind environments)
- Forward environment info to agent `record_transition` method
- Expose and document the random seeding mechanism
- Define rewards shaping function in agents' config
- Define learning rate scheduler in agents' config
- Improve agent's algorithm description in documentation (PPO and TRPO at the moment)
### Changed
- Compute the Generalized Advantage Estimation (GAE) in agent `_update` method
- Move noises definition to `resources` folder
- Update the Isaac Gym examples
### Removed
- `compute_functions` for computing the GAE from memory base class
## [0.4.1] - 2022-03-22
### Added
- Examples of all Isaac Gym environments (preview 3)
- Tensorboard file iterator for data post-processing
### Fixed
- Init and evaluate agents in ParallelTrainer
## [0.4.0] - 2022-03-09
### Added
- CEM, SARSA and Q-learning agents
- Tabular model
- Parallel training using multiprocessing
- Isaac Gym utilities
### Changed
- Initialize agents in a separate method
- Change the name of the `networks` argument to `models`
### Fixed
- Reset environments after post-processing
## [0.3.0] - 2022-02-07
### Added
- DQN and DDQN agents
- Export memory to files
- Postprocessing utility to iterate over memory files
- Model instantiator utility to allow fast development
- More examples and contents in the documentation
### Fixed
- Clip actions using the whole space's limits
## [0.2.0] - 2022-01-18
### Added
- First official release