Reward Constrained Policy Optimization

Introduction

Applying Reinforcement Learning (RL) is generally a hard problem. At each state, the agent performs an action which produces a reward. The goal is to maximize the accumulated reward, hence the reward signal implicitly defines the behavior of the agent. While in computer games (e.g. Bellemare et al. (2013)) there exists a pre-defined reward signal, it is not such in many real applications.

An example is the Mujoco domain (Todorov et al., 2012), in which the goal is to learn to control robotic agents in tasks such as: standing up, walking, navigation and more. Considering the Humanoid domain, the agent is a 3 dimensional humanoid and the task is to walk forward as far as possible (without falling down) within a fixed amount of time. Naturally, a reward is provided based on the forward velocity in order to encourage a larger distance; however, additional reward signals are provided in order to guide the agent, for instance a bonus for staying alive, a penalty for energy usage and a penalty based on the force of impact between the feet and the floor (which should encourage less erratic behavior). Each signal is multiplied by it’s own coefficient, which controls the emphasis placed on it.

This approach is a multi-objective problem (Mannor and Shimkin, 2004); in which for each set of penalty coefficients, there exists a different, optimal solution, also known as Pareto optimality (Van Moffaert and Nowé, 2014). In practice, the exact coefficient is selected through a time consuming and a computationally intensive process of hyper-parameter tuning. As our experiments show, the coefficient is not shared across domains, a coefficient which leads to a satisfying behavior on one domain may lead to catastrophic failure on the other (issues also seen in Leike et al. (2017) and Mania et al. (2018)). Constraints are a natural and consistent approach, an approach which ensures a satisfying behavior without the need for manually selecting the penalty coefficients.

In constrained optimization, the task is to maximize a target function $f(x)$ while satisfying an inequality constraint $g(x)\leq\alpha$. While constraints are a promising solution to ensuring a satisfying behavior, existing methods are limited in the type of constraints they are able to handle and the algorithms that they may support - they require a parametrization of the policy (policy gradient methods) and propagation of the constraint violation signal over the entire trajectory (e.g. Prashanth and Ghavamzadeh (2016)). This poses an issue, as Q-learning algorithms such as DQN (Mnih et al., 2015) do not learn a parametrization of the policy, and common Actor-Critic methods (e.g. (Schulman et al., 2015a; Mnih et al., 2016; Schulman et al., 2017)) build the reward-to-go based on an N-step sample and a bootstrap update from the critic.

In this paper, we propose the ‘Reward Constrained Policy Optimization’ (RCPO) algorithm. RCPO incorporates the constraint as a penalty signal into the reward function. This penalty signal guides the policy towards a constraint satisfying solution. We prove that RCPO converges almost surely, under mild assumptions, to a constraint satisfying solution (Theorem 2). In addition; we show, empirically on a toy domain and six robotics domains, that RCPO results in a constraint satisfying solution while demonstrating faster convergence and improved stability (compared to the standard constraint optimization methods).

Related work: Constrained Markov Decision Processes (Altman, 1999) are an active field of research. CMDP applications cover a vast number of topics, such as: electric grids (Koutsopoulos and Tassiulas, 2011), networking (Hou and Zhao, 2017), robotics (Chow et al., 2015; Gu et al., 2017; Achiam et al., 2017; Dalal et al., 2018) and finance (Krokhmal et al., 2002; Tamar et al., 2012).

The main approaches to solving such problems are (i) Lagrange multipliers (Borkar, 2005; Bhatnagar and Lakshmanan, 2012), (ii) Trust Region (Achiam et al., 2017), (iii) integrating prior knowledge (Dalal et al., 2018) and (iv) manual selection of the penalty coefficient (Tamar and Mannor, 2013; Levine and Koltun, 2013; Peng et al., 2018).

Novelty: The novelty of our work lies in the ability to tackle (1) general constraints (both discounted sum and mean value constraints), not only constraints which satisfy the recursive Bellman equation (i.e, discounted sum constraints) as in previous work. The algorithm is (2) reward agnostic. That is, invariant to scaling of the underlying reward signal, and (3) does not require the use of prior knowledge. A comparison with the different approaches is provided in Table 1.

Preliminaries

An important property of the value function is that it solves the recursive Bellman equation:

The goal is then to maximize the expectation of the reward-to-go, given the initial state distribution $\mu$:

2 Constrained MDPs

A Constrained Markov Decision Process (CMDP) extends the MDP framework by introducing a penalty $c(s,a)$, a constraint $C(s_{t})=F(c(s_{t},a_{t}),...,c(s_{N},a_{N}))$ and a threshold $\alpha\in$. A constraint may be a discounted sum (similar to the reward-to-go), the average sum and more (see Altman (1999) for additional examples). Throughout the paper we will refer to the collection of these constraints as general constraints.

We denote the expectation over the constraint by:

3 Parametrized Policies

In this work we consider parametrized policies, such as neural networks. The parameters of the policy are denoted by $\theta$ and a parametrized policy as $\pi_{\theta}$. We make the following assumptions in order to ensure convergence to a constraint satisfying policy:

The value $V^{\pi}_{R}(s)$ is bounded for all policies $\pi\in\Pi$.

Every local minima of $J_{C}^{\pi_{\theta}}$ is a feasible solution.

Assumption 2 is the minimal requirement in order to ensure convergence, given a general constraint, of a gradient algorithm to a feasible solution. Stricter assumptions, such as convexity, may ensure convergence to the optimal solution; however, in practice constraints are non-convex and such assumptions do not hold.

Constrained Policy Optimization

Constrained MDP’s are often solved using the Lagrange relaxation technique (Bertesekas, 1999). In Lagrange relaxation, the CMDP is converted into an equivalent unconstrained problem. In addition to the objective, a penalty term is added for infeasibility, thus making infeasible solutions sub-optimal. Given a CMDP (3), the unconstrained problem is

where $L$ is the Lagrangian and $\lambda\geq 0$ is the Lagrange multiplier (a penalty coefficient). Notice, as $\lambda$ increases, the solution to (4) converges to that of (3). This suggests a two-timescale approach: on the faster timescale, $\theta$ is found by solving (4), while on the slower timescale, $\lambda$ is increased until the constraint is satisfied. The goal is to find a saddle point $(\theta^{*}(\lambda^{*}),\lambda^{*})$ of (4), which is a feasible solution.

A feasible solution of the CMDP is a solution which satisfies $J^{\pi}_{C}\leq\alpha$.

We assume there isn’t access to the MDP itself, but rather samples are obtained via simulation. The simulation based algorithm for the constrained optimization problem (3) is:

where $\Gamma_{\theta}$ is a projection operator, which keeps the iterate $\theta_{k}$ stable by projecting onto a compact and convex set. $\Gamma_{\lambda}$ projects $\lambda$ into the range $[0,\lambda_{\text{max}}]$When Assumption 2 holds, $\lambda_{\text{max}}$ can be set to $\infty$.. $\nabla_{\theta}L$ and $\nabla_{\lambda}L$ are derived from (4), where the formulation for $\nabla_{\theta}L$ is derivied using the log-likelihood trick (Williams, 1992):

$\eta_{1}(k),\eta_{2}(k)$ are step-sizes which ensure that the policy update is performed on a faster timescale than that of the penalty coefficient $\lambda$.

Under Assumption 3, as well as the standard stability assumption for the iterates and bounded noise (Borkar et al., 2008), the iterates $(\theta_{n},\lambda_{n})$ converge to a fixed point (a local minima) almost surely.

Under assumptions 1 and 2, the fixed point of Theorem 1 is a feasible solution.

The proof to Theorem 1 is provided in Appendix C and to Lemma 1 in Appendix D.

Reward Constrained Policy Optimization

Recently there has been a rise in the use of Actor-Critic based approaches, for example: A3C (Mnih et al., 2016), TRPO (Schulman et al., 2015a) and PPO (Schulman et al., 2017). The actor learns a policy $\pi$, whereas the critic learns the value (using temporal-difference learning - the recursive Bellman equation). While the original use of the critic was for variance reduction, it also enables training using a finite number of samples (as opposed to Monte-Carlo sampling).

Our goal is to tackle general constraints (Section 2.2), as such, they are not ensured to satisfy the recursive property required to train a critic.

2 Penalized reward functions

We overcome this issue by training the actor (and critic) using an alternative, guiding, penalty - the discounted penalty. The appropriate assumptions under which the process converges to a feasible solution are provided in Theorem 2. It is important to note that; in order to ensure constraint satisfaction, $\lambda$ is still optimized using Monte-Carlo sampling on the original constraint (8).

The value of the discounted (guiding) penalty is defined as:

The penalized reward functions are defined as:

As opposed to (4), for a fixed $\pi$ and $\lambda$, the penalized value (11) can be estimated using TD-learning critic. We denote a three-timescale (Constrained Actor Critic) process, in which the actor and critic are updated following (11) and $\lambda$ is updated following (5), as the ‘Reward Constrained Policy Optimization’ (RCPO) algorithm. Algorithm 1 illustrates such a procedure and a full RCPO Advantage-Actor-Critic algorithm is provided in Appendix A.

Denote by $\Theta=\{\theta:J_{C}^{\pi_{\theta}}\leq\alpha\}$ the set of feasible solutions and the set of local-minimas of $J_{C_{\gamma}}^{\pi_{\theta}}$ as $\Theta_{\gamma}$. Assuming that $\Theta_{\gamma}\subseteq\Theta$ then the ‘Reward Constrained Policy Optimization’ (RCPO) algorithm converges almost surely to a fixed point $\left(\theta^{*}(\lambda^{*},v^{*}),v^{*}(\lambda^{*}),\lambda^{*}\right)$ which is a feasible solution (e.g. $\theta^{*}\in\Theta$).

The proof to Theorem 2 is provided in Appendix E.

The assumption in Theorem 2 demands a specific correlation between the guiding penalty signal $C_{\gamma}$ and the constraint $C$. Consider a robot with an average torque constraint. A policy which uses 0 torque at each time-step is a feasible solution and in turn is a local minimum of both $J_{C}$ and $J_{C_{\gamma}}$. If such a policy is reachable from any $\theta$ (via gradient descent), this is enough in order to provide a theoretical guarantee such that $J_{C_{\gamma}}$ may be used as a guiding signal in order to converge to a fixed-point, which is a feasible solution.

Experiments

We test the RCPO algorithm in various domains: a grid-world, and 6 tasks in the Mujoco simulator (Todorov et al., 2012). The grid-world serves as an experiment to show the benefits of RCPO over the standard Primal-Dual approach (solving (4) using Monte-Carlo simulations), whereas in the Mujoco domains we compare RCPO to reward shaping, a simpler (yet common) approach, and show the benefits of an adaptive approach to defining the cost value.

While we consider mean value constraints (robotics experiments) and probabilistic constraints (i.e., Mars rover), discounted sum constraints can be immediately incorporated into our setup. We compare our approach with relevant baselines that can support these constraints. Discounted sum approaches such as Achiam et al. (2017) and per-state constraints such as Dalal et al. (2018) are unsuitable for comparison given the considered constraints. See Table 1 for more details.

For clarity, we provide exact details in Appendix B (architecture and simulation specifics).

1.2 Experiment Description

As this domain is characterized by a discrete action space, we solve it using the A2C algorithm (a synchronous version of A3C (Mnih et al., 2016)). We compare RCPO, using the discounted penalty $C_{\gamma}$, with direct optimization of the Lagrange dual form (4).

1.3 Experiment Analysis

Figure 2 illustrates the domain and the policies the agent has learned based on different safety requirements. Learning curves are provided in Figure 2. The experiments show that, for both scenarios $\alpha=0.01$ and $\alpha=0.5$, RCPO is characterized by faster convergence (improved sample efficiency) and lower variance (a stabler learning regime).

2 Robotics

2.2 Experiment Description

In the following experiments; the aim is to prolong the motor life of the various robots, while still enabling the robot to perform the task at hand. To do so, the robot motors need to be constrained from using high torque values. This is accomplished by defining the constraint $C$ as the average torque the agent has applied to each motor, and the per-state penalty $c(s,a)$ becomes the amount of torque the agent decided to apply at each time step. We compare RCPO to the reward shaping approach, in which the different values of $\lambda$ are selected apriori and remain constant.

2.3 Experiment Analysis

Learning curves are provided in Figure 3 and the final values in Table 2. It is important to note that by preventing the agent from using high torque levels (limit the space of admissible policies), the agent may only be able to achieve a sub-optimal policy. RCPO aims to find the best performing policy given the constraints; that is, the policy that achieves maximal value while at the same time satisfying the constraints. Our experiments show that:

In all domains, RCPO finds a feasible (or near feasible) solution, and, besides the Walker2d-v2 domain, exhibits superior performance when compared to the relevant reward shaping variants (constant $\lambda$ values resulting in constraint satisfaction).

Selecting a constant coefficient $\lambda$ such that the policy satisfies the constraint is not a trivial task, resulting in different results across domains (Achiam et al., 2017).

2.4 The Drawbacks of Reward Shaping

When performing reward shaping (selecting a fixed $\lambda$ value), the experiments show that in domains where the agent attains a high value, the penalty coefficient is required to be larger in order for the solution to satisfy the constraints. However, in domains where the agent attains a relatively low value, the same penalty coefficients can lead to drastically different behavior - often with severely sub-optimal solutions (e.g. Ant-v2 compared to Swimmer-v2).

Additionally, in RL, the value ($J_{R}^{\pi}$) increases as training progresses, this suggests that a non-adaptive approach is prone to converge to sub-optimal solutions; when the penalty is large, it is plausible that at the beginning of training the agent will only focus on constraint satisfaction and ignore the underlying reward signal, quickly converging to a local minima.

Discussion

We introduced a novel constrained actor-critic approach, named ‘Reward Constrained Policy Optimization’ (RCPO). RCPO uses a multi-timescale approach; on the fast timescale an alternative, discounted, objective is estimated using a TD-critic; on the intermediate timescale the policy is learned using policy gradient methods; and on the slow timescale the penalty coefficient $\lambda$ is learned by ascending on the original constraint. We validate our approach using simulations on both grid-world and robotics domains and show that RCPO converges in a stable and sample efficient manner to a constraint satisfying policy.

An exciting extension of this work is the combination of RCPO with CPO (Achiam et al., 2017). As they consider the discounted penalty, our guiding signal, it might be possible to combine both approaches. Such an approach will be able to solve complex constraints while enjoying feasibility guarantees during training.

Acknowledgements

The authors would like to thank Nadav Merlis for the insightful discussions and helpful remarks during the writing process.

References

Appendix A RCPO Algorithm

Appendix B Experiment details

In order to avoid the issue of exploration in this domain, we employ a linearly decaying random restart [Kakade and Langford, 2002]. $\mu$, the initial state distribution, follows the following rule:

where $S$ denotes all the non-terminal states in the state space and $s^{*}$ is the state at the top left corner (red in Figure 2). Initially the agent starts at a random state, effectively improving the exploration and reducing convergence time. As training progresses, with increasing probability, the agent starts at the top left corner, the state which we test against.

The A2C architecture is the standard non-recurrent architecture, where the actor and critic share the internal representation and only hold a separate final projection layer. The input is fully-observable, being the whole grid. The network is as follows:

between the layers we apply a ReLU non-linearity.

As performance is noisy on such risk-sensitive environments, we evaluated the agent every 5120 episodes for a length of 1024 episodes. To reduce the initial convergence time, we start $\lambda$ at 0.6 and use a learning rate $lr_{\lambda}=0.000025$.

B.2 Robotics

For these experiments we used a PyTorch [Paszke et al., 2017] implementation of PPO [Kostrikov, 2018]. Notice that as in each domain the state represents the location and velocity of each joint, the number of inputs differs between domains. The network is as follows:

where DiagGaussian is a multivariate Gaussian distribution layer which learns a mean (as a function of the previous layers output) and std, per each motor, from which the torque is sampled. Between each layer, a Tanh non-linearity is applied.

We report the online performance of the agent and run each test for a total of 1M samples. In these domains we start $\lambda$ at 0 and use a learning rate $lr_{\lambda}=5e-7$ which decays at a rate of $\kappa=(1-1e-9)$ in order to avoid oscillations.

The simulations were run using Generalized Advantage Estimation [Schulman et al., 2015b] with coefficient $\tau=0.95$ and discount factor $\gamma=0.99$.

Appendix C Proof of Theorem 1

We provide a brief proof for clarity. We refer the reader to Chapter 6 of Borkar et al. for a full proof of convergence for two-timescale stochastic approximation processes.

Initially, we assume nothing regarding the structure of the constraint as such $\lambda_{\text{max}}$ is given some finite value. The special case in which Assumption 2 holds is handled in Lemma 1.

The proof of convergence to a local saddle point of the Lagrangian (4) contains the following main steps:

Convergence of $\theta$-recursion: We utilize the fact that owing to projection, the $\theta$ parameter is stable. We show that the $\theta$-recursion tracks an ODE in the asymptotic limit, for any given value of $\lambda$ on the slowest timescale.

Convergence of $\lambda$-recursion: This step is similar to earlier analysis for constrained MDPs. In particular, we show that $\lambda$-recursion in (4) converges and the overall convergence of $(\theta_{k},\lambda_{k})$ is to a local saddle point $(\theta^{*}(\lambda^{*},\lambda^{*})$ of $L(\lambda,\theta)$.

Step 1: Due to the timescale separation, we can assume that the value of $\lambda$ (updated on the slower timescale) is constant. As such it is clear that the following ODE governs the evolution of $\theta$:

where $\Gamma_{\theta}$ is a projection operator which ensures that the evolution of the ODE stays within the compact and convex set $\Theta:=\Pi_{i=1}^{k}\left[\theta_{\text{min}}^{i},\theta_{\text{max}}^{i}\right]$.

As $\lambda$ is considered constant, the process over $\theta$ is:

Thus (6) can be seen as a discretization of the ODE (12). Finally, using the standard stochastic approximation arguments from Borkar et al. concludes step 1.

Step 2: We start by showing that the $\lambda$-recursion converges and then show that the whole process converges to a local saddle point of $L(\lambda,\theta)$.

The process governing the evolution of $\lambda$:

where $\theta(\lambda_{k})$ is the limiting point of the $\theta$-recursion corresponding to $\lambda_{k}$, can be seen as the following ODE:

Finally, as seen in Theorem 2 of Chapter 2 of Borkar et al. , $\theta_{n}\rightarrow\theta^{*}$ a.s. then $\lambda_{n}\rightarrow\lambda(\theta^{*})$ a.s. which completes the proof.

Appendix D Proof of Lemma 1

The proof is obtained by a simple extension to that of Theorem 1. Assumption 2 states that any local minima $\pi_{\theta}$ of 2 satisfies the constraints, e.g. $J_{C}^{\pi_{\theta}}\leq\alpha$; additionally, Lee et al. show that first order methods such as gradient descent, converge almost surely to a local minima (avoiding saddle points and local maxima). Hence for $\lambda_{\text{max}}=\infty$ (unbounded Lagrange multiplier), the process converges to a fixed point $(\theta^{*}(\lambda^{*}),\lambda^{*})$ which is a feasible solution.

Appendix E Proof of Theorem 2

As opposed to Theorem 1, in this case we are considering a three-timescale stochastic approximation scheme (the previous Theorem considered two-timescales). The proof is similar in essence to that of Prashanth and Ghavamzadeh .

The full process is described as follows:

Step 1: The value $v_{k}$ runs on the fastest timescale, hence it observes $\theta$ and $\lambda$ as static. As the TD operator is a contraction we conclude that $v_{k}\rightarrow v(\lambda,\theta)$.

Step 2: For the policy recursion $\theta_{k}$, due to the timescale differences, we can assume that the critic $v$ has converged and that $\lambda$ is static. Thus as seen in the proof of Theorem 1, $\theta_{k}$ converges to the fixed point $\theta(\lambda,v)$.

Step 3: As shown previously (and in Prashanth and Ghavamzadeh ), $(\lambda_{n},\theta_{n},v_{n})\rightarrow(\lambda(\theta^{*}),\theta^{*},v(\theta^{*}))$ a.s.

Denoting by $\Theta=\{\theta:J_{C}^{\pi_{\theta}}\leq\alpha\}$ the set of feasible solutions and the set of local-minimas of $J_{C_{\gamma}}^{\pi_{\theta}}$ as $\Theta_{\gamma}$. We recall the assumption stated in Theorem 2:

Given that the assumption above holds, we may conclude that for $\lambda_{\text{max}}\rightarrow\infty$, the set of stationary points of the process are limited to a sub-set of feasible solutions of (4). As such the process converges a.s. to a feasible solution.

We finish by providing intuition regarding the behavior in case the assumptions do not hold.

Assumption 2 does not hold: As gradient descent algorithms descend until reaching a (local) stationary point. In such a scenario, the algorithm is only ensured to converge to some stationary solution, yet said solution is not necessarily a feasible one.

As such we can only treat the constraint as a regularizing term for the policy in which $\lambda_{\text{max}}$ defines the maximal regularization allowed.

Assumption 4 does not hold: In this case, it is not safe to assume that the gradient of (2) may be used as a guide for solving (3). A Monte-Carlo approach may be used (as seen in Section 5.1) to approximate the gradients, however this does not enjoy the benefits of reduced variance and smaller samples (due to the lack of a critic).