Soft Actor-Critic (SAC)

SAC is a model-free, stochastic off-policy actor-critic algorithm that uses double Q-learning (like TD3) and entropy regularization to maximize a trade-off between exploration and exploitation

Paper: Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

Algorithm implementation

Main notation/symbols:

- policy function approximator (\(\pi_\theta\)), critic function approximator (\(Q_\phi\))

- states (\(s\)), actions (\(a\)), rewards (\(r\)), next states (\(s'\)), dones (\(d\))

- log probabilities (\(logp\)), entropy coefficient (\(\alpha\))

- loss (\(L\))

Learning algorithm (_update(...))

# sample a batch from memory

[\(s, a, r, s', d\)] \(\leftarrow\) states, actions, rewards, next_states, dones of size batch_size

# gradient steps

FOR each gradient step up to gradient_steps DO

# compute target values

\(a',\; logp' \leftarrow \pi_\theta(s')\)

\(Q_{1_{target}} \leftarrow Q_{{\phi 1}_{target}}(s', a')\)

\(Q_{2_{target}} \leftarrow Q_{{\phi 2}_{target}}(s', a')\)

\(Q_{_{target}} \leftarrow \text{min}(Q_{1_{target}}, Q_{2_{target}}) - \alpha \; logp'\)

\(y \leftarrow r \;+\) discount_factor \(\neg d \; Q_{_{target}}\)

# compute critic loss

\(Q_1 \leftarrow Q_{\phi 1}(s, a)\)

\(Q_2 \leftarrow Q_{\phi 2}(s, a)\)

\(L_{Q_\phi} \leftarrow 0.5 \; (\frac{1}{N} \sum_{i=1}^N (Q_1 - y)^2 + \frac{1}{N} \sum_{i=1}^N (Q_2 - y)^2)\)

# optimization step (critic)

reset \(\text{optimizer}_\phi\)

\(\nabla_{\phi} L_{Q_\phi}\)

\(\text{clip}(\lVert \nabla_{\phi} \rVert)\) with grad_norm_clip

step \(\text{optimizer}_\phi\)

# compute policy (actor) loss

\(a,\; logp \leftarrow \pi_\theta(s)\)

\(Q_1 \leftarrow Q_{\phi 1}(s, a)\)

\(Q_2 \leftarrow Q_{\phi 2}(s, a)\)

\(L_{\pi_\theta} \leftarrow \frac{1}{N} \sum_{i=1}^N (\alpha \; logp - \text{min}(Q_1, Q_2))\)

# optimization step (policy)

reset \(\text{optimizer}_\theta\)

\(\nabla_{\theta} L_{\pi_\theta}\)

\(\text{clip}(\lVert \nabla_{\theta} \rVert)\) with grad_norm_clip

step \(\text{optimizer}_\theta\)

# entropy learning

IF learn_entropy is enabled THEN

# compute entropy loss

\({L}_{entropy} \leftarrow - \frac{1}{N} \sum_{i=1}^N (log(\alpha) \; (logp + \alpha_{Target}))\)

# optimization step (entropy)

reset \(\text{optimizer}_\alpha\)

\(\nabla_{\alpha} {L}_{entropy}\)

step \(\text{optimizer}_\alpha\)

# compute entropy coefficient

\(\alpha \leftarrow e^{log(\alpha)}\)

# update target networks

\({\phi 1}_{target} \leftarrow\) polyak \({\phi 1} + (1 \;-\) polyak \() {\phi 1}_{target}\)

\({\phi 2}_{target} \leftarrow\) polyak \({\phi 2} + (1 \;-\) polyak \() {\phi 2}_{target}\)

# update learning rate

IF there is a learning_rate_scheduler THEN

step \(\text{scheduler}_\theta (\text{optimizer}_\theta)\)

step \(\text{scheduler}_\phi (\text{optimizer}_\phi)\)

Configuration and hyperparameters

skrl.agents.torch.sac.sac.SAC_DEFAULT_CONFIG

SAC_DEFAULT_CONFIG = {
    "gradient_steps": 1,            # gradient steps
    "batch_size": 64,               # training batch size

    "discount_factor": 0.99,        # discount factor (gamma)
    "polyak": 0.005,                # soft update hyperparameter (tau)

    "actor_learning_rate": 1e-3,    # actor learning rate
    "critic_learning_rate": 1e-3,   # critic learning rate
    "learning_rate_scheduler": None,        # learning rate scheduler class (see torch.optim.lr_scheduler)
    "learning_rate_scheduler_kwargs": {},   # learning rate scheduler's kwargs (e.g. {"step_size": 1e-3})

    "state_preprocessor": None,             # state preprocessor class (see skrl.resources.preprocessors)
    "state_preprocessor_kwargs": {},        # state preprocessor's kwargs (e.g. {"size": env.observation_space})

    "random_timesteps": 0,          # random exploration steps
    "learning_starts": 0,           # learning starts after this many steps

    "grad_norm_clip": 0,            # clipping coefficient for the norm of the gradients

    "learn_entropy": True,          # learn entropy
    "entropy_learning_rate": 1e-3,  # entropy learning rate
    "initial_entropy_value": 0.2,   # initial entropy value
    "target_entropy": None,         # target entropy

    "rewards_shaper": None,         # rewards shaping function: Callable(reward, timestep, timesteps) -> reward

    "experiment": {
        "base_directory": "",       # base directory for the experiment
        "experiment_name": "",      # experiment name
        "write_interval": 250,      # TensorBoard writing interval (timesteps)

        "checkpoint_interval": 1000,        # interval for checkpoints (timesteps)
        "store_separately": False,          # whether to store checkpoints separately

        "wandb": False,             # whether to use Weights & Biases
        "wandb_kwargs": {}          # wandb kwargs (see https://docs.wandb.ai/ref/python/init)
    }
}

Spaces and models

The implementation supports the following Gym spaces / Gymnasium spaces

Gym/Gymnasium spaces	Observation	Action
Discrete	\(\square\)	\(\square\)
Box	\(\blacksquare\)	\(\blacksquare\)
Dict	\(\blacksquare\)	\(\square\)

The implementation uses 1 stochastic and 4 deterministic function approximators. These function approximators (models) must be collected in a dictionary and passed to the constructor of the class under the argument models

Notation	Concept	Key	Input shape	Output shape	Type
\(\pi_\theta(s)\)	Policy (actor)	"policy"	observation	action	Gaussian / MultivariateGaussian
\(Q_{\phi 1}(s, a)\)	Q1-network (critic 1)	"critic_1"	observation + action	1	Deterministic
\(Q_{\phi 2}(s, a)\)	Q2-network (critic 2)	"critic_2"	observation + action	1	Deterministic
\(Q_{{\phi 1}_{target}}(s, a)\)	Target Q1-network	"target_critic_1"	observation + action	1	Deterministic
\(Q_{{\phi 2}_{target}}(s, a)\)	Target Q2-network	"target_critic_2"	observation + action	1	Deterministic

Support for advanced features is described in the next table

Feature	Support and remarks
Shared model	-
RNN support	RNN, LSTM, GRU and any other variant

API