Noises

Definition of the noises used by the agents during the exploration stage. All noises inherit from a base class that defines a uniform interface

Implemented noises

Gaussian noise
Ornstein-Uhlenbeck noise

Basic usage

The noise usage is defined in each agent’s configuration dictionary. A noise instance is set under the "noise" sub-key. The following examples show how to set the noise for an agent:

from skrl.resources.noises.torch import GaussianNoise

cfg = DEFAULT_CONFIG.copy()
cfg["exploration"]["noise"] = GaussianNoise(mean=0, std=0.2, device="cuda:0")

from skrl.resources.noises.torch import OrnsteinUhlenbeckNoise

cfg = DEFAULT_CONFIG.copy()
cfg["exploration"]["noise"] = OrnsteinUhlenbeckNoise(theta=0.15, sigma=0.2, base_scale=1.0, device="cuda:0")

Gaussian noise

API

class skrl.resources.noises.torch.gaussian.GaussianNoise(mean: float, std: float, device: Optional[Union[str, torch.device]] = None)

Bases: skrl.resources.noises.torch.base.Noise

__init__(mean: float, std: float, device: Optional[Union[str, torch.device]] = None) → None

Class representing a Gaussian noise

Parameters

mean (float) – Mean of the normal distribution
std (float) – Standard deviation of the normal distribution
device (str or torch.device, optional) – Device on which a torch tensor is or will be allocated (default: None). If None, the device will be either "cuda:0" if available or "cpu"

Example:

>>> noise = GaussianNoise(mean=0, std=1)

sample(size: Union[Tuple[int], torch.Size]) → torch.Tensor

Sample a Gaussian noise

Parameters: size (tuple or list of integers, or torch.Size) – Shape of the sampled tensor
Returns: Sampled noise
Return type: torch.Tensor

Example:

>>> noise.sample((3, 2))
tensor([[-0.4901,  1.3357],
        [-1.2141,  0.3323],
        [-0.0889, -1.1651]], device='cuda:0')

>>> x = torch.rand(3, 2, device="cuda:0")
>>> noise.sample(x.shape)
tensor([[0.5398, 1.2009],
        [0.0307, 1.3065],
        [0.2082, 0.6116]], device='cuda:0')

sample_like(tensor: torch.Tensor) → torch.Tensor

Sample a noise with the same size (shape) as the input tensor

This method will call the sampling method as follows .sample(tensor.size())

Parameters: tensor (torch.Tensor) – Input tensor used to determine output tensor size (shape)
Returns: Sampled noise
Return type: torch.Tensor

Example:

>>> x = torch.rand(3, 2, device="cuda:0")
>>> noise.sample_like(x)
tensor([[-0.0423, -0.1325],
        [-0.0639, -0.0957],
        [-0.1367,  0.1031]], device='cuda:0')

Ornstein-Uhlenbeck noise

class skrl.resources.noises.torch.ornstein_uhlenbeck.OrnsteinUhlenbeckNoise(theta: float, sigma: float, base_scale: float, mean: float = 0, std: float = 1, device: Optional[Union[str, torch.device]] = None)

Bases: skrl.resources.noises.torch.base.Noise

__init__(theta: float, sigma: float, base_scale: float, mean: float = 0, std: float = 1, device: Optional[Union[str, torch.device]] = None) → None

Class representing an Ornstein-Uhlenbeck noise

Parameters

theta (float) – Factor to apply to current internal state
sigma (float) – Factor to apply to the normal distribution
base_scale (float) – Factor to apply to returned noise
mean (float, optional) – Mean of the normal distribution (default: 0.0)
std (float, optional) – Standard deviation of the normal distribution (default: 1.0)
device (str or torch.device, optional) – Device on which a torch tensor is or will be allocated (default: None). If None, the device will be either "cuda:0" if available or "cpu"

Example:

>>> noise = OrnsteinUhlenbeckNoise(theta=0.1, sigma=0.2, base_scale=0.5)

sample(size: Union[Tuple[int], torch.Size]) → torch.Tensor

Sample an Ornstein-Uhlenbeck noise

Parameters: size (tuple or list of integers, or torch.Size) – Shape of the sampled tensor
Returns: Sampled noise
Return type: torch.Tensor

Example:

>>> noise.sample((3, 2))
tensor([[-0.0452,  0.0162],
        [ 0.0649, -0.0708],
        [-0.0211,  0.0066]], device='cuda:0')

>>> x = torch.rand(3, 2, device="cuda:0")
>>> noise.sample(x.shape)
tensor([[-0.0540,  0.0461],
        [ 0.1117, -0.1157],
        [-0.0074,  0.0420]], device='cuda:0')

sample_like(tensor: torch.Tensor) → torch.Tensor

Sample a noise with the same size (shape) as the input tensor

This method will call the sampling method as follows .sample(tensor.size())

Parameters: tensor (torch.Tensor) – Input tensor used to determine output tensor size (shape)
Returns: Sampled noise
Return type: torch.Tensor

Example:

>>> x = torch.rand(3, 2, device="cuda:0")
>>> noise.sample_like(x)
tensor([[-0.0423, -0.1325],
        [-0.0639, -0.0957],
        [-0.1367,  0.1031]], device='cuda:0')

Base class

Note

This is the base class for all the other classes in this module. It provides the basic functionality for the other classes. It is not intended to be used directly.

Basic inheritance usage

from typing import Union, Tuple

import torch

from skrl.resources.noises.torch import Noise     # from . import Noise


class CustomNoise(Noise):
    def __init__(self, device: Union[str, torch.device] = "cuda:0") -> None:
        """
        :param device: Device on which a torch tensor is or will be allocated (default: "cuda:0")
        :type device: str or torch.device, optional
        """
        super().__init__(device)

    def sample(self, size: Union[Tuple[int], torch.Size]) -> torch.Tensor:
        """Sample noise

        :param size: Shape of the sampled tensor
        :type size: tuple or list of integers, or torch.Size

        :return: Sampled noise
        :rtype: torch.Tensor
        """
        # ================================
        # - sample noise
        # ================================

API

class skrl.resources.noises.torch.base.Noise(device: Optional[Union[str, torch.device]] = None)

Bases: object

__init__(device: Optional[Union[str, torch.device]] = None) → None

Base class representing a noise

Parameters: device (str or torch.device, optional) – Device on which a torch tensor is or will be allocated (default: None). If None, the device will be either "cuda:0" if available or "cpu"

Custom noises should override the sample method:

import torch
from skrl.resources.noises.torch import Noise

class CustomNoise(Noise):
    def __init__(self, device=None):
        super().__init__(device)

    def sample(self, size):
        return torch.rand(size, device=self.device)

sample(size: Union[Tuple[int], torch.Size]) → torch.Tensor

Noise sampling method to be implemented by the inheriting classes

Parameters: size (tuple or list of integers, or torch.Size) – Shape of the sampled tensor
Raises: NotImplementedError – The method is not implemented by the inheriting classes
Returns: Sampled noise
Return type: torch.Tensor

sample_like(tensor: torch.Tensor) → torch.Tensor

Sample a noise with the same size (shape) as the input tensor

This method will call the sampling method as follows .sample(tensor.size())

Parameters: tensor (torch.Tensor) – Input tensor used to determine output tensor size (shape)
Returns: Sampled noise
Return type: torch.Tensor

Example:

>>> x = torch.rand(3, 2, device="cuda:0")
>>> noise.sample_like(x)
tensor([[-0.0423, -0.1325],
        [-0.0639, -0.0957],
        [-0.1367,  0.1031]], device='cuda:0')