Guided Uncertainty-Aware Policy Optimization: Combining Learning and Model-Based Strategies for Sample-Efficient Policy Learning

Michelle A. Lee, Carlos Florensa, Jonathan Tremblay, Nathan Ratliff, Animesh Garg, Fabio Ramos, Dieter Fox

I INTRODUCTION

Modern robots rely on extensive systems to accomplish a given task, such as a perception module to monitor the state of the world . Simple perception failure in this context is catastrophic for the robot, since its motion generator relies on it. Moreover, classic motion generators are quite rigid in how they accomplish a task, e.g., the robot has to pick an object in a specific way and might not recover if the grasp fails. These problems make robotics systems unstable, and hard to scale to new domains. In order to expand robotics reach we need more robust, adaptive, and flexible systems.

Learning-based method, such as Reinforcement Learning (RL) has the capacity to adapt, and deal directly with raw sensory inputs, which are not subject to estimation errors . The strength of RL stems from its capacity to define a task at a higher level through a reward function indicating what to do, not through an explicit set of control actions describing how the task should be performed. RL does not need specific physical modelling as they implicitly learn a data-driven model from interacting with the environment , allowing the method to be deployed in different settings. These characteristics are desired but come with different limitations: 1) randomly interacting with an environment can be quite unsafe for the human users as well as for the equipment, 2) RL is not recognized for being sample efficient. As such, introducing RL to a new environment can be time consuming and difficult.

Classic robotic approaches have mastered generating movements within free space, where there are no contacts with other elements in the environment . We refer to these accessible methods as Model Based (MB) methods. One of their main limitations is that they normally do not handle perception errors and physical interactions naturally, e.g., grasping an object, placing an object, object insertion, etc. As such this limits the expressiveness of roboticists and the reliability of the system.

In this work we present an algorithmic framework that is aware of its own uncertainty in the perception and actuation system. As such a MB guides the agent to the relevant region, hence reducing the area where the RL policy needs to be optimized and making it more invariant to the absolute goal location. Our novel algorithm combines the strengths from MB and RL. We leverage the efficiency of MB to move in free-space, and the capacity of RL to learn from its environment from a loosely defined goal. In order to efficiently fuse MB and RL, we introduce a perception system that provides uncertainty estimates of the region where contacts might occur. This uncertainty is used to determine the region where the MB method shouldn’t be applied, and an RL policy should be learned instead. Therefore, we call our algorithm Guided Uncertainty Aware Policy Optimization (GUAPO).

Figure LABEL:fig:real shows an overview of our system, the task is initialized with MB where it guides the robot within the range of the uncertainties of the object of interest, e.g., the box where to insert the peg. Once we have reached that region, we switch to RL to complete the task. At learning time, we leverage information from task completion by the RL policy to reduce our perception system’s uncertainties. This work makes the following contributions:

We demonstrate that GUAPO outperforms pure RL, pure MB, as well as a Residual policy baseline that combines MB and RL for peg insertion;

We present a simple and yet efficient way to express pose uncertainties for a keypoint based pose estimator;

We show that our approach is sample efficient for learning methods on real-world robots.

II DEFINITIONS AND FORMULATION

III METHOD

In this section, we describe the different components of GUAPO. We define $\hat{\mathcal{S}}_{\textit{u}}$ as a super-set of $\mathcal{S}_{\textit{u}}$ generated from the perception system uncertainty estimation. We use this set to partition the space into the regions where the MB method is used, and regions where the RL policy is trained. Then we describe a MB method that can now confidently be used outside of $\hat{\mathcal{S}}_{\textit{u}}$ to bring the robot within that set. Finally we define the RL policy, and how the learning can be more efficient by making its inputs local. We also outline our algorithm in Algorithm 1.

Coarse perception systems are usually cheaper and faster to setup because they might require simpler hardware like RGB cameras, and can be used out-of-the-box without excessive tuning and calibration efforts . If we use such a system to directly localize $\mathcal{S}_{\textit{u}}$ , the perception errors might misleadingly indicate that a certain area belongs to $\mathcal{S}_{\textit{free}}$ , hence trying to apply the MB method and potentially not being able to learn how to recover from there. Instead, we propose to use a perception system that also gives an uncertainty estimate. Many methods can represent the uncertainty by a nonparametric distribution, with $n$ possible descriptions of the region $\{\mathcal{S}^{i}_{\textit{u}}\}_{i=1}^{n}$ and their associated weights $p(\mathcal{S}^{i}_{\textit{u}})$ . By interpreting these weights as the likelihoods $P(\mathcal{S}^{i}_{\textit{u}}=\mathcal{S}_{\textit{u}})$ , we can express the likelihood of a certain state $s$ belonging to $\mathcal{S}_{\textit{u}}$ as:

Therefore we use a switch between these two policies, based on the uncertainty estimate. A lower perception uncertainty reduces the area where the reinforcement learning method is required, and improves the overall efficiency. We now detail how each of these policies is obtained.

III-B Model-based actuation

In the previous section we defined $\hat{\mathcal{S}}_{\textit{u}}$ , the region containing the goal set $\mathcal{S}_{g}$ and hence the agent’s reward. In our problem statement we assume that outside that region, the environment model is well known, and therefore it is amenable to use a model-based approach. Therefore, whenever we are outside of $\hat{\mathcal{S}}_{\textit{u}}$ , the MB approach corrects any deviations.

Our formulation can be extended for obstacle avoidance. Using a similar approach used to over-estimate the set $\hat{\mathcal{S}}_{\textit{u}}$ , we can over-estimate the obstacle set to be avoided by $\hat{\mathcal{S}}^{\textit{obst}}_{\textit{u}}$ , and remove that space from where the MB method can be applied, $\mathcal{S}_{\textit{free}}$ . An obstacle-avoiding MB method can be used to get to the area where the goal is, while avoiding the regions where the obstacle might be, as shown in our videoshttps://sites.google.com/view/guapo-rl.

III-C From Model-Based to Reinforcement learning

Once $\pi_{\textit{MB}}$ has brought the system within $\hat{\mathcal{S}}_{\textit{u}}$ , the control is handed-over to $\pi_{\textit{RL}}$ as expressed in Eq. 2. Note that our switching definition goes both ways, and therefore if $\pi_{\textit{RL}}$ takes exploratory actions that move it outside of $\hat{\mathcal{S}}_{\textit{u}}$ , the MB method will act again to funnel the state to the area of interest. This also provides a framework for safe learning in case there are obstacles to avoid as introduced in the section above. There are several advantages to having a more restricted area where the RL policy needs to learn how to act: first the exploration becomes easier, second, the policy can be local. In particular, we only feed to $\pi_{\textit{RL}}$ the images from a wrist-mounted camera and its current velocities, as depicted in Fig. 1(b). Not using global information from our perception system in Fig. 1(a) can make our RL policy generalize better across locations of $\hat{\mathcal{S}}_{\textit{u}}$ . Finally, we propose to use an off-policy RL algorithm, so all the observed transitions can be added in the replay buffer, no matter if they come from $\pi_{\textit{MB}}$ or $\pi_{\textit{RL}}$ .

III-D Closing the MB-RL loop

This framework also allows to use any newly acquired experience to reduce $\hat{\mathcal{S}}_{\textit{u}}$ such that successive rollouts can use the model-based method in larger areas of the state-space. For example, in the peg-insertion task, once the reward of fully inserting the peg is received, the location of the opening is immediately known. Since we no longer need to rely on the noisy perception system to estimate the location of the hole, we can update $\hat{\mathcal{S}}_{\textit{u}}=\mathcal{S}_{\textit{u}}$ , where now the reinforcement learning algorithm only needs to do the actual insertion and not keep looking for the opening.

IV IMPLEMENTATION DETAILS

Here we describe the implementation details of our GUAPO algorithm for a peg insertion task with a Franka Panda robot (7-DoF torque-controlled robot). We first introduce the perception module and how an uncertainty estimate is obtained to localize $\hat{\mathcal{S}}_{\textit{u}}$ . Then we describe the model-based policy used to navigate in $\mathcal{S}_{\textit{free}}$ while avoiding obstacles, and the RL algorithm and the architecture of the RL policy being learned. Finally, we introduce our task set-up, the baseline algorithms we compare GUAPO with, and their implementations.

We use Deep Object Pose Estimator (DOPE) as the base for our perception system. DOPE uses a simple neural network architecture that can be quickly trained with synthetic data and domain randomization using NDDS . Figure 2 shows generated images with domain randomization used to train our perception system and thus allowing domain transfer (from synthetic to real world). Note that the model of the object that DOPE needs to detect is not very detailed, consisting of the approximate shape without texture. This is a challenging case, specially because no depth sensing is used to supplement the RGB information. DOPE algorithm first finds the object cuboid keypoints using local peaks on the map. Using the cuboid real dimensions, camera intrinsics, and the keypoint locations, DOPE runs a PnP algorithm to find the final object pose in the camera frame.

For this work we extended the DOPE perception system to obtain uncertainty estimates of the object pose. This extension augments the peak estimation algorithm by fitting a 2d Gaussian around each found peak, as depicted by the dark contour maps in Fig. 1(a). We then run PnP algorithm on $n$ set of keypoints, where each set of keypoint is constructed by sampling from all the 2d Gaussians. This provides $n$ possible poses of the object consistent with the detection algorithm, as drawn in green bounding boxes in Fig. 1(a). In this work we treat them as equally likely.

From our problem formulation, we assume access to a rough description of the area of interest, $\mathcal{S}_{\textit{u}}$ , around the object where an operation needs to be performed. In our peg insertion task, this is a rectangle centered at the opening of the hole. For each of the $n$ pose samples given by our extended DOPE perception algorithm, we compute the associated hole opening positions, $\{h_{i}\}_{i=1}^{n}$ , represented by the green dots in Fig. 1(a). These points are then fitted by 3d Gaussian with diagonal covariance, represented in blue in the same figure. We use the mean $\hat{\mu}_{\textit{hole}}=\frac{1}{n}\sum_{i=1}^{n}h_{i}$ as the center of $\hat{\mathcal{S}}_{\textit{u}}$ and we over-approximate Eqn. 1 by displacing $\mathcal{S}_{\textit{u}}$ along the axis by one standard deviation.

The perception module setup is depicted in Fig. LABEL:fig:real in orange, where the camera for DOPE (640x480x3 RGB images from Logitech Carl Zeiss Tessar) is mounted overlooking our workspace. The top center image with the orange border is a sample from that camera.

IV-B Model-Based Controller Design

The policies are sending end-effector position commands at 20 Hz. The RMPs are computing desired joint positions $\mathbf{q}_{d}$ at 1000 Hz. Given that impedance-end-effector control is an action space which has been shown to improve sample efficiency for policy learning for RL , we also use the RMPs interface as our reinforcement learning action space.

IV-C Reinforcement Learning Algorithm and Architecture

We use a state-of-the-art model-free off-policy RL algorithm, Soft Actor Critic . The RL policy acts directly from raw sensory inputs. This consists on joint velocities and images from a wrist-mounted camera (64x64x3 RGB images from a Logitech Carl Zeiss Tessar) on the robot (see Fig. LABEL:fig:real). As illustrated in Fig. 1(b), all inputs are fed into a $\beta$ -VAE . The VAE gives us a low-dimensional latent-space representation of the state, which has been shown to improve sample efficiency of RL algorithms . The parameters of this VAE are trained before-hand on a data-set collected off-line. The only part that is learned by the RL algorithm is a 2-layer MLP that takes as input the 64-dimensional latent representation given by the VAE, and produces 3D position displacement $\Delta\mathbf{x}$ of the robot end-effector.

IV-D Training Details

The VAE is pre-trained with 160,000 datapoints for 12 epochs, on the Titan XP GPU. DOPE is trained for 8 hours on 4 p100 GPU. All our learning-based policy methods (GUAPO, SAC baseline, and the Residual Policy baseline described in Sec. V) were trained for 60 training iterations. In total, each policy was trained with 120 training episodes, as each iteration has two training episodes, each with 1000 steps. This takes 90 min. to train.

IV-E Rewards

For GUAPO, we use a sparse reward when the policy finishes the task (inserts the peg). The policy gets -1 everywhere, and 0 when it finishes the task. For our other learning-based baselines (SAC and Residual policy ), we use a negative L2 norm to the perception estimate of the goal location $\hat{\mu}_{\textit{hole}}$ , 0 when it reaches $\hat{\mathcal{S}}_{\textit{u}}$ , and 1 when it finishes the task.

V EXPERIMENTAL DESIGN AND RESULTS

In this section we seek to answer the following questions: How does our method compares to our baseline policies, such as, Residual policies, in terms of sample efficiency and task completion? And, is the proposed algorithm capable of performing peg insertion on a real robot?

All the different baselines were initialized about 75 cm away from the goal. They were all implemented on our real robotics system. As such we compare our proposed method to the following:

MB-Perfect. This method consists of a scripted policy under perfect state estimation.

MB-Rand-Perfect. This method uses the same policy as MB-Perfect where we injected random actions, which we sample from a normal distribution with 0 mean and a standard deviation defined by the perception uncertainty from DOPE (which is around 2.5cm to 3 cm).

MB-DOPE. This method is similar to MB-Perfect, but instead uses the pose estimator prediction to servo to the hole and accomplish insertion.

MB-Rand-Dope. This method uses the same policy as MB-Dope where we injected random actions, which is sampled in the same way as MB-Rand-Perfect.

SAC. This uses just the policy learned from the RL algorithm, Soft-Actor Critic (SAC), to accomplish the task.

Residual. This method is based off recent residual-learning techniques that combine model-based and reinforcement learning methods .

V-B Results

The results comparing the different methods is shown in Table I, this table presents the success rate for insertion as well as the average number of steps needed for completion (a step is equivalent to 50 milliseconds of following the same robot command, as our policy is running at 20 Hz), and the percentage that the end-effector ends up in the $\mathcal{S}_{\textit{u}}$ and $\hat{\mathcal{S}}_{\textit{u}}$ regions over 30 trials. We also present training iteration performance (task success and steps to completion) for the different methods in Figure 3.

MB-Perfect is able to insert 100% of the time, as it has perfect knowledge of the state, and can be seen as an oracle. We can see that taking random actions with MB-Rand-Perfect does not degrade excessively the full performance achieved by MB-Perfect. However, when we used DOPE as the perception system, which has around 2.5 to 3.5 cm of noise and error, the performance of MB-DOPE and MB-Rand-DOPE drops drastically. MB-Rand-DOPE performs 26.6% better than MB-DOPE, as the random actions can help offset the perception error.

In our setup SAC did not achieve any insertion. This is due to the low number of samples that SAC trained on, since most success stories of RL in the real world require several orders of magnitude more data . The Residual method also did not achieve any insertions. The Residual method often would apply large actions far away from the hole opening, and end up sliding off the box and getting stuck pushing against the side of the box. In comparison, GUAPO only turns on the reinforcement learning policy once it is already nearby the region of interest, and hence does not suffer from this. However, Residual was able to reach $\hat{\mathcal{S}}_{\textit{u}}$ 100% of the time after 120 training episodes, while SAC never did.

In comparison, as seen in Fig. 3, after around 8 training iterations, GUAPO is also able to start inserting into the hole (which is about 12 minute real-world training time). As the policy trains, the average number of steps it takes to insert the peg also decreases. After 120 training episodes (and 90 minutes of training), GUAPO is able to achieve 93% insertion rate.

VI RELATED WORK

In robotic manipulation there are two dominating paradigms to perform a task: leveraging model of the environment (model-based method) or leveraging data to learn (learning-based method). The first category of methods relies on a precise description of the task, such as object CAD models, as well as powerful and sophisticated perception systems . With an accurate model, a well engineered solution can be designed for that particular task , or the model can then be combined with some search algorithm like motion planning . This type of model-based approach is limited by the ingenuity of the roboticist, and could lead to irrecoverable failure if the perception system has un-modeled noise and error.

On the other hand, learning-based approaches in manipulation do not require such detailed description, but rather require access to interaction with the environment, as well as a reward that indicates success. Such binary rewards are easy to describe, but unfortunately they render Reinforcement Learning methods extremely sample-inefficient. Hence many prior works use shaped rewards , which requires considerable tuning. Other works use low-dimensional state spaces instead of image inputs, which requires either precise perception systems or specially-designed hardware with sensors. There are some proposed methods that manage to deal directly with the sparse rewards, like automatic curriculum generation or the use of demonstrations , but these approaches still require large amounts of interaction with the environment. Furthermore, if the position of the objects in the scene changes or there are new distractors in the scene, these methods need to be fully retrained. On the other hand, our method is extremely sample-efficient with a sparse success reward, and is robust to these variations thanks to the model-based component.

Recent works can also be understood as combining model-based and learning-based approaches. One such method uses a reinforcement learning algorithm to find the best parameters that describe the behavior of the agent based on a model-based template. The learning is very efficient, but at the cost of an extremely engineered pre-solution that also relies on an accurate perception system. Another line of work that allows to combine model-based and learning-based methods is Residual Learning , where RL is used to learn an additive policy that can potentially fully over-write the original model-based policy and does not require any further structure. Nevertheless, these methods are hard to tune, and hardly preserve any of the benefits of the underlying model-based method once trained.

The problem of known object pose estimation is a vibrant subject within the robotics and computer vision communities . Regressing to keypoints on the object or on a cuboid encompassing the object seems to have become the defacto approach for the problem. Keypoints are first detected by a neural network, then P $n$ P is used to predict the pose of the object. Peng et al. also explored the problem of using uncertainty by leveraging a ransac voting algorithm to find regions where a keypoint could be detected. This approach differs from ours as they do not directly regress to a keypoint probability map, they regress to a vector voting map, where line intersection is then used to find keypoints. Moreover their method does not carry pose uncertainty in the final prediction.

VII CONCLUSIONS

We introduce a novel algorithm, Guided Uncertainty Aware Policy Optimization (GUAPO), that combines the generalization capabilities of model-based methods and the adaptability of learning-based methods. It allows to loosely define the task to perform, by solely providing a coarse model of the objects, and a rough description of the area where some operation needs to be performed. The model-based system leverage this high-level information and accessible state estimation systems to create a funnel around the area of interest. We use the uncertainty estimate provided by the perception system to automatically switch between the model-based policy, and a learning-based policy that can learn from an easy-to-define sparse reward, overcoming the model and estimation errors of the model-based part. We show learning in the real world of a peg insertion task.

ACKNOWLEDGMENT

Carlos Florensa and Michelle Lee are grateful to all the robotics team at NVIDIA for providing a great learning environment, and providing constant support. Special thanks to Ankur Handa for helping with the compute infrastructure.