Random memory
Basic usage
1import torch 2from skrl.memories.torch import RandomMemory 3 4# create a random memory object 5memory = RandomMemory(memory_size=1000, num_envs=1, replacement=False) 6 7# create tensors in memory 8memory.create_tensor(name="states", size=(64, 64, 3), dtype=torch.float32) 9memory.create_tensor(name="actions", size=(4,1), dtype=torch.float32) 10memory.create_tensor(name="rewards", size=1, dtype=torch.float32) 11memory.create_tensor(name="next_states", size=(64, 64, 3), dtype=torch.float32) 12memory.create_tensor(name="dones", size=1, dtype=torch.bool) 13 14# add data to the memory 15for i in range(500): 16 memory.add_samples(states=torch.rand(64, 64, 3).view(-1), 17 actions=torch.rand(4,1).view(-1), 18 rewards=torch.rand(1), 19 next_states=torch.rand(64, 64, 3).view(-1), 20 dones=torch.randint(2, size=(1,)).bool()) 21 22# sample a batch of data from the memory 23batch = memory.sample(batch_size=32, names=["states", "actions", "rewards", "next_states", "dones"])
API
- class skrl.memories.torch.random.RandomMemory(memory_size: int, num_envs: int = 1, device: Optional[Union[str, torch.device]] = None, export: bool = False, export_format: str = 'pt', export_directory: str = '', replacement=True)
Bases:
skrl.memories.torch.base.Memory
- __init__(memory_size: int, num_envs: int = 1, device: Optional[Union[str, torch.device]] = None, export: bool = False, export_format: str = 'pt', export_directory: str = '', replacement=True) None
Random sampling memory
Sample a batch from memory randomly
- Parameters
memory_size (int) – Maximum number of elements in the first dimension of each internal storage
num_envs (int, optional) – Number of parallel environments (default: 1)
device (str or torch.device, optional) – Device on which a torch tensor is or will be allocated (default:
None
). If None, the device will be either"cuda:0"
if available or"cpu"
export (bool, optional) – Export the memory to a file (default: False). If True, the memory will be exported when the memory is filled
export_format (str, optional) – Export format (default: “pt”). Supported formats: torch (pt), numpy (np), comma separated values (csv)
export_directory (str, optional) – Directory where the memory will be exported (default: “”). If empty, the agent’s experiment directory will be used
replacement (bool, optional) – Flag to indicate whether the sample is with or without replacement (default: True). Replacement implies that a value can be selected multiple times (the batch size is always guaranteed). Sampling without replacement will return a batch of maximum memory size if the memory size is less than the requested batch size
- Raises
ValueError – The export format is not supported
- __len__() int
Compute and return the current (valid) size of the memory
The valid size is calculated as the
memory_size * num_envs
if the memory is full (filled). Otherwise, thememory_index * num_envs + env_index
is returned- Returns
Valid size
- Return type
- add_samples(**tensors: torch.Tensor) None
Record samples in memory
Samples should be a tensor with 2-components shape (number of environments, data size). All tensors must be of the same shape
According to the number of environments, the following classification is made:
one environment: Store a single sample (tensors with one dimension) and increment the environment index (second index) by one
number of environments less than num_envs: Store the samples and increment the environment index (second index) by the number of the environments
number of environments equals num_envs: Store the samples and increment the memory index (first index) by one
- Parameters
tensors (dict) – Sampled data as key-value arguments where the keys are the names of the tensors to be modified. Non-existing tensors will be skipped
- Raises
ValueError – No tensors were provided or the tensors have incompatible shapes
- create_tensor(name: str, size: Union[int, Tuple[int], gym.spaces.space.Space, gymnasium.spaces.space.Space], dtype: Optional[torch.dtype] = None, keep_dimensions: bool = False) bool
Create a new internal tensor in memory
The tensor will have a 3-components shape (memory size, number of environments, size). The internal representation will use _tensor_<name> as the name of the class property
- Parameters
name (str) – Tensor name (the name has to follow the python PEP 8 style)
size (int, tuple or list of integers, gym.Space, or gymnasium.Space) – Number of elements in the last dimension (effective data size). The product of the elements will be computed for sequences or gym/gymnasium spaces
dtype (torch.dtype or None, optional) – Data type (torch.dtype). If None, the global default torch data type will be used (default)
keep_dimensions (bool) – Whether or not to keep the dimensions defined through the size parameter (default: False)
- Raises
ValueError – The tensor name exists already but the size or dtype are different
- Returns
True if the tensor was created, otherwise False
- Return type
- get_sampling_indexes() Union[tuple, numpy.ndarray, torch.Tensor]
Get the last indexes used for sampling
- Returns
Last sampling indexes
- Return type
tuple or list, numpy.ndarray or torch.Tensor
- get_tensor_by_name(name: str, keepdim: bool = True) torch.Tensor
Get a tensor by its name
- Parameters
- Raises
KeyError – The tensor does not exist
- Returns
Tensor
- Return type
- get_tensor_names() Tuple[str]
Get the name of the internal tensors in alphabetical order
- Returns
Tensor names without internal prefix (_tensor_)
- Return type
tuple of strings
- load(path: str) None
Load the memory from a file
Supported formats: - PyTorch (pt) - NumPy (npz) - Comma-separated values (csv)
- Parameters
path (str) – Path to the file where the memory will be loaded
- Raises
ValueError – If the format is not supported
- reset() None
Reset the memory by cleaning internal indexes and flags
Old data will be retained until overwritten, but access through the available methods will not be guaranteed
Default values of the internal indexes and flags
filled: False
env_index: 0
memory_index: 0
- sample(names: Tuple[str], batch_size: int, mini_batches: int = 1, sequence_length: int = 1) List[List[torch.Tensor]]
Sample a batch from memory randomly
- Parameters
- Returns
Sampled data from tensors sorted according to their position in the list of names. The sampled tensors will have the following shape: (batch size, data size)
- Return type
list of torch.Tensor list
- sample_all(names: Tuple[str], mini_batches: int = 1, sequence_length: int = 1) List[List[torch.Tensor]]
Sample all data from memory
- Parameters
- Returns
Sampled data from memory. The sampled tensors will have the following shape: (memory size * number of environments, data size)
- Return type
list of torch.Tensor list
- sample_by_index(names: Tuple[str], indexes: Union[tuple, numpy.ndarray, torch.Tensor], mini_batches: int = 1) List[List[torch.Tensor]]
Sample data from memory according to their indexes
- Parameters
names (tuple or list of strings) – Tensors names from which to obtain the samples
indexes (tuple or list, numpy.ndarray or torch.Tensor) – Indexes used for sampling
mini_batches (int, optional) – Number of mini-batches to sample (default: 1)
- Returns
Sampled data from tensors sorted according to their position in the list of names. The sampled tensors will have the following shape: (number of indexes, data size)
- Return type
list of torch.Tensor list
- save(directory: str = '', format: str = 'pt') None
Save the memory to a file
Supported formats:
PyTorch (pt)
NumPy (npz)
Comma-separated values (csv)
- Parameters
- Raises
ValueError – If the format is not supported
- set_tensor_by_name(name: str, tensor: torch.Tensor) None
Set a tensor by its name
- Parameters
name (str) – Name of the tensor to set
tensor (torch.Tensor) – Tensor to set
- Raises
KeyError – The tensor does not exist
Share the tensors between processes