Twin-Delayed DDPG (TD3)

TD3 is a model-free, deterministic off-policy actor-critic algorithm (based on DDPG) that relies on double Q-learning, target policy smoothing and delayed policy updates to address the problems introduced by overestimation bias in actor-critic algorithms

Paper: Addressing Function Approximation Error in Actor-Critic Methods

Algorithm implementation

Main notation/symbols:

- policy function approximator (\(\mu_\theta\)), critic function approximator (\(Q_\phi\))

- states (\(s\)), actions (\(a\)), rewards (\(r\)), next states (\(s'\)), dones (\(d\))

- loss (\(L\))

Decision making (act(...))

\(a \leftarrow \mu_\theta(s)\)

\(noise \leftarrow\) sample noise

\(scale \leftarrow (1 - \text{timestep} \;/\) timesteps \() \; (\) initial_scale \(-\) final_scale \() \;+\) final_scale

\(a \leftarrow \text{clip}(a + noise * scale, {a}_{Low}, {a}_{High})\)

Learning algorithm (_update(...))

# sample a batch from memory

[\(s, a, r, s', d\)] \(\leftarrow\) states, actions, rewards, next_states, dones of size batch_size

# gradient steps

FOR each gradient step up to gradient_steps DO

# target policy smoothing

\(a' \leftarrow \mu_{\theta_{target}}(s')\)

\(noise \leftarrow \text{clip}(\) smooth_regularization_noise \(, -c, c) \qquad\) with \(c\) as smooth_regularization_clip

\(a' \leftarrow a' + noise\)

\(a' \leftarrow \text{clip}(a', {a'}_{Low}, {a'}_{High})\)

# compute target values

\(Q_{1_{target}} \leftarrow Q_{{\phi 1}_{target}}(s', a')\)

\(Q_{2_{target}} \leftarrow Q_{{\phi 2}_{target}}(s', a')\)

\(Q_{_{target}} \leftarrow \text{min}(Q_{1_{target}}, Q_{2_{target}})\)

\(y \leftarrow r \;+\) discount_factor \(\neg d \; Q_{_{target}}\)

# compute critic loss

\(Q_1 \leftarrow Q_{\phi 1}(s, a)\)

\(Q_2 \leftarrow Q_{\phi 2}(s, a)\)

\(L_{Q_\phi} \leftarrow \frac{1}{N} \sum_{i=1}^N (Q_1 - y)^2 + \frac{1}{N} \sum_{i=1}^N (Q_2 - y)^2\)

# optimization step (critic)

reset \(\text{optimizer}_\phi\)

\(\nabla_{\phi} L_{Q_\phi}\)

\(\text{clip}(\lVert \nabla_{\phi} \rVert)\) with grad_norm_clip

step \(\text{optimizer}_\phi\)

# delayed update

IF it’s time for the policy_delay update THEN

# compute policy (actor) loss

\(a \leftarrow \mu_\theta(s)\)

\(Q_1 \leftarrow Q_{\phi 1}(s, a)\)

\(L_{\mu_\theta} \leftarrow - \frac{1}{N} \sum_{i=1}^N Q_1\)

# optimization step (policy)

reset \(\text{optimizer}_\theta\)

\(\nabla_{\theta} L_{\mu_\theta}\)

\(\text{clip}(\lVert \nabla_{\theta} \rVert)\) with grad_norm_clip

step \(\text{optimizer}_\theta\)

# update target networks

\(\theta_{target} \leftarrow\) polyak \(\theta + (1 \;-\) polyak \() \theta_{target}\)

\({\phi 1}_{target} \leftarrow\) polyak \({\phi 1} + (1 \;-\) polyak \() {\phi 1}_{target}\)

\({\phi 2}_{target} \leftarrow\) polyak \({\phi 2} + (1 \;-\) polyak \() {\phi 2}_{target}\)

# update learning rate

IF there is a learning_rate_scheduler THEN

step \(\text{scheduler}_\theta (\text{optimizer}_\theta)\)

step \(\text{scheduler}_\phi (\text{optimizer}_\phi)\)

Configuration and hyperparameters

skrl.agents.torch.td3.td3.TD3_DEFAULT_CONFIG

TD3_DEFAULT_CONFIG = {
    "gradient_steps": 1,            # gradient steps
    "batch_size": 64,               # training batch size

    "discount_factor": 0.99,        # discount factor (gamma)
    "polyak": 0.005,                # soft update hyperparameter (tau)

    "actor_learning_rate": 1e-3,    # actor learning rate
    "critic_learning_rate": 1e-3,   # critic learning rate
    "learning_rate_scheduler": None,        # learning rate scheduler class (see torch.optim.lr_scheduler)
    "learning_rate_scheduler_kwargs": {},   # learning rate scheduler's kwargs (e.g. {"step_size": 1e-3})

    "state_preprocessor": None,             # state preprocessor class (see skrl.resources.preprocessors)
    "state_preprocessor_kwargs": {},        # state preprocessor's kwargs (e.g. {"size": env.observation_space})

    "random_timesteps": 0,          # random exploration steps
    "learning_starts": 0,           # learning starts after this many steps

    "grad_norm_clip": 0,            # clipping coefficient for the norm of the gradients

    "exploration": {
        "noise": None,              # exploration noise
        "initial_scale": 1.0,       # initial scale for the noise
        "final_scale": 1e-3,        # final scale for the noise
        "timesteps": None,          # timesteps for the noise decay
    },

    "policy_delay": 2,                      # policy delay update with respect to critic update
    "smooth_regularization_noise": None,    # smooth noise for regularization
    "smooth_regularization_clip": 0.5,      # clip for smooth regularization

    "rewards_shaper": None,         # rewards shaping function: Callable(reward, timestep, timesteps) -> reward

    "experiment": {
        "directory": "",            # experiment's parent directory
        "experiment_name": "",      # experiment name
        "write_interval": 250,      # TensorBoard writing interval (timesteps)

        "checkpoint_interval": 1000,        # interval for checkpoints (timesteps)
        "store_separately": False,          # whether to store checkpoints separately

        "wandb": False,             # whether to use Weights & Biases
        "wandb_kwargs": {}          # wandb kwargs (see https://docs.wandb.ai/ref/python/init)
    }
}

Spaces and models

The implementation supports the following Gym spaces / Gymnasium spaces

Gym/Gymnasium spaces	Observation	Action
Discrete	\(\square\)	\(\square\)
Box	\(\blacksquare\)	\(\blacksquare\)
Dict	\(\blacksquare\)	\(\square\)

The implementation uses 6 deterministic function approximators. These function approximators (models) must be collected in a dictionary and passed to the constructor of the class under the argument models

Notation	Concept	Key	Input shape	Output shape	Type
\(\mu_\theta(s)\)	Policy (actor)	"policy"	observation	action	Deterministic
\(\mu_{\theta_{target}}(s)\)	Target policy	"target_policy"	observation	action	Deterministic
\(Q_{\phi 1}(s, a)\)	Q1-network (critic 1)	"critic_1"	observation + action	1	Deterministic
\(Q_{\phi 2}(s, a)\)	Q2-network (critic 2)	"critic_2"	observation + action	1	Deterministic
\(Q_{{\phi 1}_{target}}(s, a)\)	Target Q1-network	"target_critic_1"	observation + action	1	Deterministic
\(Q_{{\phi 2}_{target}}(s, a)\)	Target Q2-network	"target_critic_2"	observation + action	1	Deterministic

Support for advanced features is described in the next table

Feature	Support and remarks
Shared model	-
RNN support	RNN, LSTM, GRU and any other variant

API