Learning High-DOF Reaching-and-Grasping via Dynamic Representation of Gripper-Object Interaction

Introduction

Robotic grasping is an important and long-standing problem in robotics. It has been drawing increasingly broader attention from the fields of computer vision (Saxena et al., 2010), machine learning (Kleeberger et al., 2020) and computer graphics (Liu, 2009; Pollard and Zordan, 2005), due to the interdisciplinary nature of its core techniques. Traditional methods typically approach grasping by breaking the task into two stages: static grasp synthesis followed by motion planning of the gripper. In the first stage, a few candidate grasp poses are generated. The second stage then plans a collision-avoiding trajectory for the robotic arm and gripper to select a feasible grasp and execute it. The main limitation of such a decoupled approach is that the two stages are not jointly optimized which could lead to suboptimal solutions.

An integrated solution to grasp planning and motion planning, which is often referred to as the reach-and-grasp problem, remains a challenge (Wang et al., 2019). A few works have attempted integrated grasp and motion planning by formulating a joint optimization problem. The advantage of an integrated planner is that motion planning and grasp planning impose constraints on each other. However, these existing methods still rely on pre-sampled grasp candidates over which a probabilistic distribution for selection is computed, making it highly reliant on the quality of the candidates. Wang et al. (2019) introduce online grasp synthesis to eliminate the need for a perfect grasp set and grasp selection heuristics. Nevertheless, such an approach optimizes over a discrete set of grasp candidates, which limits the grasping space explored.

Reinforcement learning (RL) (Sutton and Barto, 2018) models offer a counterpoint to the planning paradigm. Rather than optimizing for grasp selection and motion planning, the idea is to use closed-loop feedback control based on sensory observations so that the agent can dynamically update its strategy while accumulating new observations. More recent advances of RL allow for continuous, high-dimensional actions which are especially suitable for continuous exploration of reach-and-grasp planning. Albeit offering promising solutions, the sample efficiency issue of RL hinders its application in highly complex control scenarios, such as dexterous grasping of a high-DOF robotic hand (e.g., a 24-DOF five-fingered gripper).

We argue that the main cause of the limitation above is the lack of an effective representation of observations. Indeed, even with deep neural networks as powerful function approximators, it is still too difficult to fit a function mapping raw sensory observations (camera images) to low-level robot actions (e.g., motor torques, velocities, or Cartesian motions). Therefore, learning complicated control of high-DOF grippers calls for an informative representation of the intermediate states during the reaching-and-grasping process. Such representation should well inform the RL model about the dynamic interaction between the gripper and the target object.

In this work, we advocate the use of Interaction Bisector Surface (IBS) for representing gripper-object interaction in RL-based reach-and-grasp learning. IBS, computed as the Voronoi diagram between two geometric objects, was originally proposed for indexing and recognizing inter-protein relation in the area of biology (Kim et al., 2006). In computer graphics, it has been successfully applied in characterizing fine-grained spatial relations between 3D objects (Zhao et al., 2014; Hu et al., 2015). We found that IBS is surprisingly effective as a state/observation representation in learning high-DOF reach-and-grasp planning. Gripper-object IBS well informs the global pose of the gripper and the fine-grained local control of each finger with spatial relation against the target object, making the map from observation to action easier to model. For different initial configuration of the gripper, i.e., different relative pose to the object, our method is able to provide different motion sequences and form different final graspings as shown in Figure 1. Moreover, during the reaching-and-grasping process, the dynamic change of relationship between the gripper and object can be well-reflected by the corresponding IBS, which enables our method to deal with moving objects with dynamic pose change, going beyond static object grasping of most previous works. In addition, as IBS is defined purely based on the geometry of the gripper and the object without any assumption of the object semantics or affordance, our method can generalize well to objects of unseen categories.

To speed up the computation of IBS for efficient training and testing, we propose a grid-based approximation of IBS as well as post-refinement mechanism to improve accuracy. Empirical studies show that the approximation is accurate to capture the important interaction information and fast enough to enable online computation in an interactive frame rate. To capture richer information of interaction, we propose a combination of local and global encoders for multi-level feature extraction of IBS, based on its segmentation corresponding to the different components of the gripper.

We adopt Soft Actor-Critic (SAC) (Haarnoja et al., 2018), an off-policy model, as our RL framework. Aside from the core design of state representation, we introduce two other critical designs to make our model more sample efficient and easier to train. To learn collision-avoiding finger motions, we impose finger-object contact information as a constraint of RL. A straightforward option is to design the reward to punish finger-object penetration. This can greatly complicate model training. Thus, our first key design is to enhance the standard scalar Q value into a vector storing finger-wise contact information. Such disentangled Q representation provides a more informative dictation on learning contact-free motions.

To deal with the continuous action space in grasp planning, we opt to learn from imperfect demonstrations synthesized offline with heuristic planning from different initial configurations to the final grasp poses generated GraspIt! (Miller and Allen, 2004). The demonstration grasping trajectories, possibly containing gripper-object penetration, are stored in the experience replay buffer (Mnih et al., 2015) of SAC. Our second key design is to bootstrap the learning with a bootstrapping replay buffer containing imperfect demonstrations possibly with collision. We then inject the replay buffer with an increasing amount of experiences sampled from the currently learned policy and rectified to be collision-free. This forms an enhanced replay buffer based on which a gradually refined policy can be learned. Such a double replay buffer scheme helps learn a strong control planning model fairly efficiently.

Our method generates high-quality dexterous grasps for unseen complex shapes with smooth grasping motions. Furthermore, our method can dynamically adjust and adapt to object movement during grasping, thus allowing to grasp moving objects. When compared to other baseline methods, our method consistently achieves a higher success rate on different datasets. Our method is also robust to partial observations of target objects. Our contributions include:

A novel state representation for learning reach-and-grasp planning based on gripper-object interaction bisector surface, along with an accurate approximation for fast training.

A combination of local and global encoders for multi-level feature extraction of IBS to capture richer information of gripper-object interaction.

A new vector-based representation of Q value encoding not only regular rewards but also finger-wise contact information for efficient learning of contact-free grasping motion.

A double replay buffer mechanism for learning collision-avoiding grasping policy from imperfect demonstrations.

Related Work

Robotic grasping has a large body of literature. Existing approaches can generally be classified into analytical and data-driven methods. Analytical (or geometric) approaches analyze the shape of the target object to synthesize a suitable grasp (Sahbani et al., 2012). Data-driven (or empirical) approaches based on machine learning are gaining increasing attention in recent years (Bohg et al., 2013; Kleeberger et al., 2020) .

With known object shapes, analytical methods search for grasp poses that maximize a certain grasp quality metric, and these methods can mainly be classified into discrete sampling-based techniques or continuous optimization techniques. Some sampling-based techniques search in the space of grasp poses (Miller and Allen, 2000, 2004), while others search contact points or contact areas on surfaces of objects and then searches for collision-free grasp poses that realize the given set of contact points or areas (Chen and Burdick, 1993; Hang et al., 2017; Liu et al., 2020a; Pan et al., 2022). Compared to sampling-based techniques, continuous optimization techniques plan grasp poses by optimizing differentiable losses (Kiatos and Malassiotis, 2019; Maldonado et al., 2010), which are more efficient. Recent work (Liu et al., 2020b) proposes a differentiable grasp quality, which can be used for continuous optimization and deep learning methods. For high-DOF grippers, however, sampling-based techniques are computationally costly due to the large search space. Continuous optimization techniques for high-DOF grasp planning apt to find local optimal grasp poses. Moreover, analytical methods are difficult to generalize to incomplete or unknown objects.

Learning-based robotic grasping.

Learning-based methods are typically split into supervised learning and reinforcement learning. For supervised learning, grasp annotations can be collected either by humans (Depierre et al., 2018), with simulation (Mahler et al., 2016), or through real robot tests (Levine et al., 2018). Supervised grasp learning can be categorized as discriminative or generative depending on whether the grasp configuration is the input or output of the learned model. While discriminative approaches sample grasp candidates and rank them using a neural network (Mahler et al., 2018), generative approaches directly generate suitable grasp poses (Morrison et al., 2018).

Early learning-based works mainly focus on generating grasps for target objects with low-DOF grippers (such as parallel-jaw grippers) (Saxena et al., 2007; Gualtieri et al., 2016). Such work learns to regress grasp quality or to predict grasp success (Mahler et al., 2016, 2017; Fang et al., 2018; Lu et al., 2020b; Lu et al., 2020a; Van der Merwe et al., 2019), but still needs sampling-based techniques to search for better grasps. Liu et al. (2019; 2020b) use deep neural networks to directly regress high-DOF grasps based on the input of voxels or depth images.

Reinforcement learning for robotic grasping.

Deep reinforcement learning (RL) has been shown as a promising and powerful technique to automatically learn control policies by trial and error. Based on raw sensory inputs, dexterous grasping behaviors can be performed. A comparative study of RL-based grasping methods is given in (Quillen et al., 2018). QT-Opt (Kalashnikov et al., 2018) learns various manipulation strategies with dynamic responses to disturbances. Song et al. (2020) present an RL-based closed-loop 6D grasping of novel objects with the help of human demonstrations. The learned policy can operate in dynamic scenes with moving objects. Rajeswaran et al. (2017) show that model-free RL can effectively scale up to complex manipulation tasks with a high-DOF hand. Mandikal and Grauman (2021) introduce an approach for learning dexterous grasps, and the key idea is to embed an object-centric visual affordance model within a deep reinforcement learning loop to learn grasping policies that favor the same object regions favored by people. Andrychowicz et al. (2020) explore RL for dexterous in-hand manipulation by reorientating a block. Starke et al. (2019) present a good example of conditioning motion behavior on the geometry of the surrounding 3D environment. Ficuciello et al. propose methods that use RL to search good grasp poses (2016) as well as good grasp motion trajectories (2019) in a synergies subspace, and these methods need good initial parameters to reduce the searching space while finding initial parameters need additional imitation learning or human efforts. To overcome this limitation, Ficuciello et al. (2019) introduce a visual module to predict initial parameters given visual information of objects. The main weakness of their methods is that the learning process should be done for every object respectively to achieve a good grasp.

Training RL models in the real environment is usually prohibitive since it requires a large number of trials and errors. A straightforward approach is to train them in a simulation environment and then transfer the learned policy to the real world with sim-to-real techniques (Peng et al., 2018; James et al., 2019). Our work trains an RL model for dexterous high-DOF grasping. We focus on how to train such complicated planning policies with the help of an effective representation of gripper-object interaction and leave the issue of sim-to-real transfer for future work.

Grasp representation.

Existing grasping models have adopted various representations to describe the shape of the target object to be grasped, such as voxels (Varley et al., 2017), depth images (Viereck et al., 2017), multi-view images (Collet and Srinivasa, 2010), or geometric primitives (Aleotti and Caselli, 2012). Some works represent a grasp with Independent Contact Regions (ICRs) (Roa and Suárez, 2009; Fontanals et al., 2014). These regions are defined such that if each finger is positioned on its corresponding contact region, a force-closure grasp (Nguyen, 1988) is always obtained, independently of the exact location of each finger.

Our work opts to characterize the interaction between the gripper and the object using Interaction Bisector Surface (IBS) (Zhao et al., 2014). Interaction Bisector Surface (IBS) captures the spatial boundary between two objects. It can provide a more detailed and informative interaction representation with both geometric and topological features extracted on the IBS. Hu et al. (2015) further combined IBS with Interaction Region (IR), which is used to describe the geometry of the surface region on the object corresponding to the interaction, to encode more geometric features on the objects. Pirk et al. (2017) build a spatial and temporal representation of interactions named interaction landscapes.

Karunratanakul et al. (2020) proposes an implicit representation to encode hand-object grasps by defining the distance to human hand and objects for points in space, and then use it to generate the grasping hand pose for a given static shape. In contrast, our method utilizes constantly updated IBS as state representation to plan the dynamic motion of the gripper for the object reach-and-grasp task.

Overview

Given the input scene segmented into foreground object and background context and the initial configuration of the gripper, our goal is to output a sequence of collision-free actions which move the gripper towards the object to perform a successful grasping. We train a deep reinforcement learning model in which the state is given by the gripper-object interaction and the actions are designed as the gripper configuration changes. Figure 2 gives an overview of one step of the planned reaching-and-grasping motion.

Our method starts by extracting an informative representation of the interaction between the given object and the current gripper configuration. A set of local and global features are extracted from the given state to predict the action that changes the configuration of the gripper so that it moves closer to the given object and forms a better grasping. The updated configuration after applying the predicted action is then passed through the same pipeline to predict the subsequent action.

We use IBS (Zhao et al., 2014) to represent the interaction. Since the computation of accurate IBS is time-consuming, we design an IBS sampler to obtain a discretized and simplified version of IBS in a much more efficient manner. The set of sampled IBS points is then moved to be closer to the exact IBS.

State encoding.

The gripper configuration is encoded by its global position and orientation as well as local joint angels with (6+18)-DOF. Based on the part components of the gripper model, i.e., five fingers and one palm, we adopt a multi-level representation of IBS inspired by the work of Zhao et al. (2017). More specifically, for each sampled IBS point, we first find its nearest points on the scene and the gripper, and then encode it with a set of information, consisting of its own coordinate, the component labels of those two nearest points as well as the spatial relationship relative to each nearest point. Therefore, the components of the gripper model, including five fingers and one palm, naturally induce a segmentation of the IBS based on the association between the gripper and the IBS. We then combine local features extracted separately for each IBS segment and global feature for the entire IBS to form a multi-level description of gripper-object interaction.

Action prediction.

To predict the action given the current state, we design a policy network that takes both local and global features from the current configuration and outputs the configuration change of the gripper and terminate value. The policy network is trained via reinforcement learning using the Soft Actor-Critic method (Haarnoja et al., 2018). To avoid gripper-scene collision and self-collision of the gripper during the reaching-and-grasping process, we design a new vector-based representation of Q value encoding not only regular rewards but also finger-wise contact information for efficient learning of contact-free grasping motion. To further accelerate the training, we generate a set of imperfect demonstration data to bootstrap the learning and to converge to the final policy in a more efficient and effective manner.

Method

The IBS is essentially the set of points equidistant from two sets of points sampled on the scene and the gripper, respectively. The computation of exact IBS requires the extraction of the Voronoi diagram, which is time-consuming. To trade-off between efficiency and accuracy, we compute IBS only within a given range and discretize the space to obtain an approximation of IBS, as shown in Figure 3.

More specifically, IBS is only computed within the sphere which is located at the centroid of the palm with a radius of $r$. The bounding box of the sphere is discretized into a $k^{3}$ grid for IBS point sampling. For each of the grid points, i.e., the center of each grid cell, we compute its distances to both the scene $d_{s}$ and the gripper $d_{g}$, and the points having $\delta=d_{g}-d_{s}=0$ are IBS points. To find such set of points, we store the $\delta$ values on grid cells and extract the zero-crossing points on the fly. Note that to accelerate the whole process, the $\delta$ values on only a partial set of cells around the exact IBS are computed in a region growing manner. More specifically, we first find the cell that is closest to the middle position of the line connecting the centroid of the foreground object in the scene and the palm, and compute its $\delta$ value. Then the computation grows outwards with the neighboring cells until sufficient grid cells that share the same sign have been found.

Figure 3 shows the sampling process of IBS in two different states. We can see that the grid is divided by IBS into two parts, i.e., one with points closer to the scene (in red color) and the other with points closer to the gripper (in blue color). Only the $\delta$ values on the grid cells close to the exact IBS are computed to get the initial set of sampled IBS points (highlighted with solid colors). Note that such set of sampled IBS contains two layers of grid points which has redundant information, thus we only keep the layer corresponding to the gripper as our IBS point set (the solid layer in blue).

Since the initial set of sampled IBS points are grid centers and the approximation accuracy is highly affected by the grid resolution, we further refine the locations of these points to make them approach the exact IBS. For each point $p$, we first locate its nearest points on scene $p_{s}$ and gripper $p_{g}$. Without loss of generality, let us assume that the distance $d_{g}$ between the point to the gripper is larger than the distance $d_{s}$ between the point and the scene. We then need to move $p$ towards $p_{g}$ to make it closer to the exact IBS: $p^{\prime}=p+\Delta_{p}\times\overrightarrow{pp_{g}}$. The detail of how to derive an appropriate $\Delta_{p}$ can be found in the supplementary material. Once the location of point $p$ is updated, we update its nearest points on the scene and on the gripper.

To gauge how close the sampled IBS is to the exact IBS as well as the effectiveness of our refinement method, we measure the IBS approximation error using the Chamfer distance between the points sampled from the grids and those sampled on the exact IBS surface. Figure 4 shows how the Chamfer distance changes during the reach-and-grasp process before and after the refinement under different grid resolutions $k=20$, $40$, and $80$. We can find several interesting properties. Firstly, approximation error is relatively stable during the reach-and-grasp process for each setting and gets slightly higher towards the end as the IBS surface becomes more complex. Secondly, approximation error drops significantly after the refinement for different grid resolutions, which shows the effectiveness of the refinement process. Thirdly, approximation error generally increases with the decrease of grid resolution, especially when no refinement is involved. However, the computation time increases with the grid resolution. To obtain a good balance, we set grid resolution $k=40$ and sample $n=4096$ points for IBS approximation.

2. State and action representation

Finding an informative representation for a given state is the key to guiding the movement of the gripper towards a successful grasping of the object. Our key observation is that the IBS between the scene and gripper together with the gripper configuration provide rich information about the intermediate state.

For gripper configuration, we use a (6+18)-DOF Shadow Hand in all our experiments, where the first 6-DOF encodes the global orientation and position of the gripper and the remaining 18-DOF encodes the joint angles. To better describe the local context around the gripper to guide its configuration change, we set the origin of the world coordinate to the centroid of the palm, as the setting in the IBS sampling process, to encode the spatial interaction features between the gripper and the scene. We found that our method gets similar performance when using the centroid of the object as the origin of the world coordinate.

For each point $p$ on the sampled IBS, we store the following information as illustrated in Figure 5:

unit vector pointing to the nearest point $p_{s}$ on the scene $v_{s}^{p}\in R^{3}$

indicator of whether $p_{s}$ is located on the foreground object $b_{s}^{p}\in\{0,1\}$

unit vector pointing to the nearest point $p_{h}$ on the gripper $v_{g}^{p}\in R^{3}$

one-hot indicator of the gripper component that $p_{g}$ belongs to $c_{g}^{p}\in\{0,1\}^{6}$

value defined on $p_{g}$ indicating which side of the gripper it located on $a_{g}^{p}\in$

Here, $a_{g}^{p}=n_{p}\cdot d_{up}$ is the dot product of the normal direction $n_{p}$ of point $p_{g}$ on gripper in rest pose and the upright direction $d_{up}$ perpendicular to the palm and pointing outwards.

Action representation.

The action consists of two parts. The first part is defined as the gripper configuration change, which is also with (6+18)-DOF. For each single action, we restrain the change of each parameter within $[-0.25cm,0.25cm]$ for global translation of the entire gripper, and $[-0.025,0.025]$ in radian for both global rotation angle of the entire gripper or local rotation angle of each joint. The other part is the stop action, defined as a termination value to indicate the possibility of terminating the planning process. The exact termination mechanism is provided in Section 4.3. .

3. Network and reward design

Figure 6 shows the network architecture of our grasp planner (actor network). For the sampled IBS, both local and global features are extracted to concatenate with the feature extracted from the gripper configuration, and then the concatenated feature is passed to two other MLPs to get the predicted action, one for the gripper configuration change and the other for the terminal value. We use PointNet (Qi et al., 2017) for both local and global IBS encoder, and the global encoder takes the whole IBS as input while the local encoder takes each IBS component corresponding to different gripper components as input.

Reward function.

The reward function needs to reflect the quality of an executed action. As our final goal is to perform a successful grasping of the given object in the scene at the end of the planning, we first define a grasp reward function $R_{g}$ to measure the grasp quality when the whole process is terminated. Moreover, to further encourage a more natural grasp pose with more gripper components taking part in the grasping and avoiding collision with the scene during the whole process, we define another reaching reward function $R_{c}^{i}$ per gripper component $g_{i}$ to provide guidance for each intermediate step.

We measure the grasp quality from two aspects. First is the commonly used execution success, which we use the success signal $S$ obtained from the Pybullet simulator when performing the final grasp. We consider the grasp to be successful if and only if the object can be lifted up by more than 0.2m as in the previous work (Xu et al., 2021). More details about the simulator setup and how to perform the grasp can be found in the supplementary material. However, this sparse boolean value cannot provide enough guidance for high-DOF grasping, thus we complement it with another well-known geometric measure $Q_{1}$ (Ferrari and Canny, 1992). But as the traditional $Q_{1}$ measure can only be computed when the gripper touches the object without any collision and the computation becomes unstable during the training, we adopt the generalized $Q_{1}$ measure proposed in (Liu et al., 2020b) instead.

As those two grasp quality metrics are only computed when the reach-and-grasp task is completed, to encourage a quick convergence, we set a negative reward $r_{f}$ for each intermediate step. So the grasp reward function is defined as:

where we set $\omega_{s}=150$, $\omega_{q}=1000$, and $r_{f}=-3$ in all our experiments. To encourage the contact between the gripper and the object, our planning process terminates requires that not only if a randomly sampled value is smaller than the terminal value but also at least two gripper components contact the object in the current step.

To determine whether a gripper component $g_{i}$ is contacting with the object, we would like to ensure that there are enough points on the gripper component that are close enough to the object while not colliding with the scene. So we first count the number $m_{i}$ of IBS points determined together by the inner side of the gripper component $g_{i}$ ($c_{n}^{p}=i,a_{g}^{p}\geq 0$) and the object ($b_{s}^{p}=1$) with distance to the object smaller than a given threshold $\delta_{d}=0.5cm$ ($d_{s}^{p}<\delta_{d}$), i.e., IBS points with $b_{s}^{p}=1,c_{n}^{p}=i,a_{h}^{p}\geq 0,d_{s}^{p}<\delta_{d}$. Then we further check if any of the IBS points determined by the gripper component $g_{i}$ (i.e., IBS points with $c_{n}^{p}=i$) is on the inner side of the scene by computing the angle between the corresponding vector $v_{s}^{p}$ pointing from the IBS point to the nearest point on the scene and the normal $n_{s}^{p}$ of the nearest point $p_{s}$. If the angle is smaller than $90$ degrees, we consider the IBS point is on the inner side of the scene, and count the number $n_{i}$ of such IBS points. If $n_{i}\geq\delta_{n}$, we consider the gripper component is colliding with the scene. Therefore, if the gripper component $g_{i}$ is not colliding with the scene ($n_{i}<\delta_{n}$) and there are enough contacting points ($m_{i}\geq\delta_{m}$), we consider the gripper component is contacting the object.

Thus we define the reaching reward function $R_{c}^{i}$ per gripper component $g_{i}$ as follows to encourage a more effective reaching-and-grasping process with more contacting but not colliding points:

In all our experiments, we set $\delta_{n}=\delta_{m}=3$ and $\eta=40$.

4. Network training

To train the network, we adopt the well-known off-policy method Soft Actor-Critic (Haarnoja et al., 2018) and make some small modifications to the Q-network to make the training more effective.

SAC has two networks, i.e., the policy network (also known as the actor network) and the Q-network (also known as the critic network). The policy network outputs a Gaussian distribution $\pi(*|s;\theta)$ to sample action for an input state $s$, where $\theta$ are the parameters of the network. Q-network outputs the evaluation value $Q(s,a;\Phi)$ for given part state $s$ and action $a$, where $\Phi$ are the parameters of the network. SAC uses an addition target Q-network to calculate target value for temporal difference (TD) update, whose parameters are denoted by $\Phi^{\prime}$. A transition is denoted as a tuple ${\{s,a,R,s^{\prime},d\}}$, where $R$ is the reward and $d$ indicates whether state $s^{\prime}$ is a terminated state. All the transitions will be stored in a replay buffer $D$ and those two networks are trained by the data sampled from $D$.

The key change to the original SAC networks is that instead of outputting a single scale value to estimate the reward, we let the Q-network output a vector $(Q_{g},Q_{c}^{0},\dots,Q_{c}^{5})$ to estimate both $R_{g}$ and $R_{c}^{i}$. Accordingly, the reward $R$ in each experience is a vector of $(R_{g},R_{c}^{0},\dots,R_{c}^{5})$. Note that only $R_{g}$ is accumulated while $R_{c}^{i}$ is computed for each single step. We found that this change made the training more stable and prevented the collision more effectively than directly combining all rewards together.

The loss function for training the Q-network is then determined by temporal difference (TD) update:

with $\lambda=0.25$ balancing the two type of rewards and target value $y_{g}$ for $R_{g}$ defined as:

where ${\widetilde{a}}^{\prime}\sim\pi(*|s^{\prime};\theta))$ is the sampled action. $\gamma=0.99$ is the discount factor, and $\alpha$ is the temperature parameter that can be adjusted automatically to match an entropy target in expectation, to balance exploring the environment and maximizing reward.

For the policy network, in order to avoid self-collision of the gripper, i.e., the collision among different gripper components, we add a self-collision loss $L_{\text{self}}$ adapted from (Liu et al., 2020b) to the original loss:

where $\omega=100$ is the parameter to balance the two loss terms. The original loss function $L_{Q}(\theta)$ and the self-collision loss $L_{\text{self}}$ for training the policy network are defined as:

where $\widetilde{a}\sim\pi(*|s;\theta))$ is the sampled action based on current state and network parameters, $L$ the number of gripper links, $N$ the number of points $p_{j}(s;\theta)$ sampled from each link, $H_{i}(s;\theta)$ the convex hull of each link after performing action $\widetilde{a}$ on state $s$, and $D$ the signed distance from a point to a convex hull. More details about the self-collision loss $L_{\text{self}}$ can be found in (Liu et al., 2020b). The reason why we add the self-collision loss directly for the policy network instead of defining a corresponding reward function is that the self-collision loss is differentiable and thus can be optimized directly.

Note that the searching space of the action is extremely large and to make the training more efficient, we adopt the popular training with demonstration strategy. To generate the demonstrations, as shown in Figure 7, we first generate a valid grasp pose for the given object, and then move the gripper away from the object along the direction pointing from the object center to the palm center until the distance reaches $d=20cm$, and then we reset the gripper configuration by setting all joint angles to be zero and add some random Gaussian noise scaled based on the rotation limit of each joint to the gripper configuration to generate a random initial configuration. After getting the initial configuration of the gripper, to generate a motion sequence towards the corresponding final grasp pose and use it as the demonstration, we simply first move the gripper to the final position and then transit to the final grasping configuration using linear interpolation.

Note that the demonstration generated in this way is imperfect since the gripper may collide with the scene during the whole process, so unlike previous methods that usually use imitation learning for behavior cloning of the perfect demonstrations, we store the generated imperfect demonstration into the bootstrapping replay buffer and use reinforcement learning only.

In more detail, we use two replay buffers, one for demonstration data with maximal size set to be $n_{d}=5.0\times 10^{4}$ and the other for self-exploration data with maximal size set to be $n_{s}=1.0\times 10^{5}$. Before training, we always fill up the demonstration buffer and keep counting the total number $n_{t}$ of data generated in those two ways. The probability of sampling data from the demonstration buffer is set to be $n_{d}/n_{t}$. Thus, more demonstration data will be sampled in the beginning to guide the network to learn a good initial policy quickly and hence speed up the training process. Following (Vecerik et al., 2017), these samples are included in the update of both the actor and the critic.

Results and Evaluation

We first explain the experiment setup, the dataset we used, and then evaluate our method both qualitatively and quantitatively.

We first adopt the dataset provided by Liu et al. (2020b), which consists of 500 objects collected from four datasets. We use the objects from KIT Dataset (2012) and GD Dataset (2015) as training data, and then test objects from YCB Dataset (2017) and BigBIRD dataset (2014). We then further test our method on more objects from ContactPose Dataset (2020) and from 3DNet Dataset (2012) when compared to baseline methods.

To generate the demonstrations to guide the training of our network, we randomly select 100 grasp poses for each object in the training set from the pose set associated with that object, and synthesize imperfect reach-and-grasp motion as described in Section 4.4. Note that since each object is placed on the table in our setting, we need to first filter out invalid grasp poses which collide with the table.

To generate our self-exploration data, we need to sample the initial configuration of the gripper. For each object, we use its center to create a sphere with radius $r=20cm$, and sample points on the upper hemisphere as the origin of the local coordinate system of the gripper and rotate the gripper to make its palm face the object center and its thumb point upwards.

As different initial configurations will lead to different reach-and-grasp results, to remove the bias of such initialization, we set a fixed set of initial configurations for each object to test. The initialization details can be found in the supplementation material.

2. Qualitative results

Figure 8 shows the gallery of results obtained with our method, where we show the initial input configuration of the gripper and four sampled frames during the approaching process with the final grasp pose on the right. Note that each example is shown with a view we selected to demonstrate the motion sequence more clearly, and we further add a purple curve to visualize moving trajectory together with sampled frames. When inspecting these results, we see that the method is able to handle various shapes given different initial configurations of the gripper. For example, for the small shark model shown in the first row, our method can adjust the gripper pose and joints precisely to pinch the model, while for the model shown in the second row, although the gripper starts from the position close to the table, our method is able to generate the moving sequence with the final grasp on the top of the model to avoid collision with the table. Our method is also able to deal with objects with more complex geometry. For example, to grasp the binoculars shown in the third row and the elephant shown in the fourth row, those four fingers tend to move together to form the gripping grasp with the thumb, while for the pitcher shown in the last row, fingers spread widely to better wrap its top.

Moreover, Figure 9 shows examples of results where we fix the shape and plan the grasping with different initial gripper configurations. For each example, we show four different initial configurations on the aligned hemisphere with purple dots on the left, and then their corresponding final grasp poses on the right. We see that various grasp poses can be generated even for the same shape when given different initial configurations. For example, for the elephant model shown in the first row, some final grasp poses tend to cover the head of the elephant while some others prefer wrapping the back of the model. Similar results can also be observed for the shoe model shown in the second row. For these models with far more complex geometry than the models we have in the training set, the final grasp poses obtained by our method can adapt well to their shapes while avoiding collision with the table at the same time. Overall, our method is quite robust and can successfully plan grasping for different shapes with various geometries when given different initial gripper configurations.

Note that all the objects tested in our experiments are unseen objects from different datasets, which shows that our method can generalize well to other objects instead of just remembering the grasp poses shown in the demonstrations. To further justify this, we compare the grasp pose synthesized by our method from the same initial configuration of the demonstration grasp in Figure 10. We can see that even starting from the same initial configuration, our method ends up with quite different and generally more natural grasp poses based on the learned policy.

3. Quantitative evaluation

To quantitatively evaluate our method, we first conduct ablation studies to justify several key design choices of our method, and then we provide comparisons to several baseline methods to show the superiority of our method.

As explained in Section 4.3, our goal is to learn a reach-and-grasp planner with high-quality final grasps as well as low penetration during the whole process, thus we use metrics to measure the final grasp and process penetration, respectively. For the final grasp, we first compute the success rate $S$ among the whole testing set, where whether each final grasp is successful or not is tested in the simulator. Then for all successful grasps, we compute the average generalized $Q_{1}$ (Liu et al., 2020b) as a complementary metric to get a more detailed grasp quality measure. For the process penetration, we further compute the percentage of the testing objects, of which the penetration of each frame during the whole reach-and-grasp process is smaller than a given threshold $\tau$, where $\tau=1mm,2mm$ and $3mm$.

Using IBS as the dynamic state representation is the key contribution of our method. Based on this, we further propose to use demonstrations, the vector-based representation of Q value, and a combination of the local and global encoder to boost the training and improve the performance. To show the importance of all those design choices, we use the simplest version of our method as the baseline, denoted as “IBS-G”, which only uses a global encoder for IBS and a standard scalar Q value and is trained without demonstrations, and then gradually add those key components one by one in the ablation studies to show how performance is boosted accordingly. To further justify the superiority of IBS as the state representation, we also compare different versions of our method with either RGBD images or point clouds as state representation. The evaluation results of the ablation studies are shown in Table 1.

Importance of demonstration. Training with demonstrations is crucial to make our network be able to learn meaningful policy. We can see in the Table 1 that without demonstration, the baseline “IBS-G” cannot learn any meaningful policy to perform valid grasping and thus the evaluation metrics cannot be reported, due to the high complexity of our problem and the corresponding large searching space. When trained with demonstration, denoted as “+ Demo”, The network is able to learn a reasonable policy now, although the performance is not satisfactory enough with only $28.7\%$ success rate. Moreover, for the most of those successful grasps, the penetration is higher than other settings.

Importance of vector-based representation of Q value. Making the Q-Network output a vector instead of a single scale value provides better control of each individual gripper component. To justify the benefit of such a design, we further enable this feature in our method and report the results in the third row of Table 1 , denoted as “+ Q-Vec”. We see that compared to the results using a scalar Q value, the performance gets highly boosted, including both success rate and penetration avoidance. The main reason is that when combining the grasp quality measure and the contacting measure together into one single value, it’s hard for the network to learn which is the main reason to cause the change of the value, while using a vector-based representation can provide a more clear guidance for the network to learn.

Importance of multi-level encoder. To further show the importance of the multi-level encoder used for feature extraction of IBS, i.e., using both global and local encoders and then concatenating the features together, we further add the local encoder to get the full version of our method, and the result is shown in the fourth row of Table 1 , denoted as “+ IBS-L”. We can see that the performance is better than other settings, which shows that the component-based IBS partition and the corresponding local encoder can help get more information and give better control of those gripper components.

Importance of IBS as state representation. The introduction of IBS to represent the intermediate state during the whole planning process and encode the interaction between the gripper and the scene is one of our key contributions. To show the importance of our IBS encoding, we compare our method to two alternative ways of interaction representation, i.e., using either RGBD images or point clouds of the scene and the gripper, while all the other design choices are the same to our method, including using demonstrations and vector-based representation of Q value.

For RGBD image presentation, we use visual information captured by a simulated hand-mounted camera in its upright-oriented pose that translates together with the hand but does not rotate. The visual input consists of an RGBD image and the corresponding segmentation information, where each pixel can belong to the object, the gripper or the background. We then replace the encoder with similar architecture to that of the network used in (Jain et al., 2019). More details about this baseline can be found in the supplementary material.

For point cloud representation, we use the scene point cloud and gripper point cloud directly as input. For the features to be encoded, we remove the features that contain interaction information from the list of features we defined for IBS points in Section 4.2 and keep all the remaining ones, which include the coordinate $c_{s}\in R^{3}$ and foreground/background indicator $b_{s}\in{0,1}$ for each scene point and coordinate $c_{g}\in R^{3}$, gripper component indicate $c_{g}\in\{0,1\}^{6}$ and gripper side indicator $a_{g}\in$ for each gripper point. The encoder is similar to that we used for IBS-G. More details about this baseline can also be found in the supplementary material.

The corresponding performances are shown in the last two rows of Table 1, denoted as “RGBD image” and “Point cloud”, respectively. Note that our method with either setting performs poorly . We think the main reason is that they cannot provide enough spatial information between the object and the gripper when the demonstrations provided are imperfect and with large variations in initial configurations and final grasp poses, while the IBS we used provides a more informative representation.

Comparison to baselines.

Strategies for the reach-and-grasp task can be roughly divided into two categories, one synthesizes the grasp first and then plans the transfer from the initial configuration to the final grasp pose and the other optimizes the whole process directly. To give more detailed comparisons to previous methods, we organize the baseline methods according to those two different types of strategies and analyze the results separately.

For the first type of baselines, we compare the final grasps obtained using our method to those synthesized using the method in (Liu et al., 2020b) and GraspIt! (Miller and Allen, 2004). For the final grasp pose generated via GraspIt!, we select the one closest to the initial gripper configuration from a pre-sampled set of candidate grasp poses based on the distance metric proposed in (Di Gregorio, 2008) To further generate the whole reach-and-grasp process, we use the same motion planning method (Kavraki et al., 1996) from The Open Motion Planning Library (OMPL) (Sucan et al., 2012) to plan a collision-free moving trajectory from the initial configuration to the global pose of the final pose. We can execute the whole process in our dynamics simulator to see if the object can be successfully grasped in the end to get an overall success. In more detail, once the gripper in the rest pose reaches its final position and orientation following the planned trajectory, we start using the interface provided by the simulator to move fingers towards the target joint states unless it contacts with the scene. When all fingers stop moving, we use the simulator to execute the grasping test. Accordingly, we can compute the success rate for different stages. The first one is “final grasp”, where we test whether the gripper with the given final configuration can successfully grasp and lift the object in the dynamics simulator. The second one is “motion plan”, where we check whether the given planner can find a feasible trajectory to transit the gripper from the initial configuration to the final configuration. The third one is “overall”, where we consider the whole process to be successful if and only if both final grasp and motion plan succeed.

Table 2 reports the success rate of the final grasp, motion plan, and the overall process executed in the dynamic simulator . Note that the success rates are computed for different settings. “Avg” refers to the average success rate among all testing samples, while “ Top-1” refers to the average success rate of all testing objects, where each object is tested with a fixed set of initial configurations and is considered to be successfully grasped if any of those initial configurations leads to a successful grasp. We can see that using the final grasp obtained via our method gets consistently better results.

Figure 11 shows some visual comparisons of those methods. For each example, we show the final grasp synthesized by each method on the left and several key frames of the reach-and-grasp process with the offline planned trajectory in purple on the right. From the results, the method of Liu et al. (2020b) cannot handle the tabletop well: It tends to generate a grasp pose with finger touching the table and does not form a valid grasp pose, especially for flat-shaped objects shown in the first row. Although GraspIt! can generate much more successful grasps, it usually fails to plan a collision-free moving path for the gripper from the given initial configuration to the final pose, and even with successfully planned path, the gripper configuration is more likely to be changed during the reaching process to avoid collision with the tabletop, which leads to a lower overall success rate. Note how the final grasp pose after the reaching process is different from the planned final poses shown in the fifth row. But for the final grasp pose generated by our method, we have taken the tabletop into the consideration during the whole reaching process, thus on the one hand, our method gets a much higher success rate of the final grasp, one the other hand, even using the same motion planning method instead of the own motion planned by our method, we can still get the best overall success rate.

For the second type of baselines, we compare the whole reach-and-grasp process obtained via our method to a heuristic primitive-based grasping method. Inspired by the work of (Della Santina et al., 2019), we adopt three grasping primitives, including “Pinch”, “Top”, and “Lateral”, and for each testing initial configuration, we select the closest primitive based on the geodesic distance on the sphere to execute the corresponding grasp. Note that “Top” and “Pinch” primitives have the same start gripper configurations, and we choose the one that achieves higher performance to get the final result. More details about this primitive-based method (PBM) can be found in the supplementary material.

Table 3 reports the comparisons of “Avg” and “ Top-1” success rate of overall execution in the simulator on four different datasets. We can see that our method outperforms the primitive-based method on all the datasets. The main reason is the reaching path of each primitive grasp is fixed and cannot adapt well to the various geometry of different objects, while our method can keep targeting the object with the guidance of the encoded IBS. Figure 12 shows some visual comparisons between our method and PBM. Note how the final grasp poses of the primitive-based method deviate from the object due to the shape complexity. For example, neither the final grasps of PBM shown in the first and third examples touches the object.

One interesting aspect of our method we would like to highlight is that when executing the reach-and-grasp process in the simulator, the pose of the target object may change due to collision with the gripper, and our method is still able to dynamically adapt to its new pose, follow the moving object and grasp it successfully. See the grasp result of the bottle shown in the fourth row of Figure 12. Other than the adaption to dynamic changes of the target object, our method is also robust to partial observations. Examples can be found in the supplementary materials.

Discussion and Future Work

We have presented an RL-based method to jointly learn grasp and motion planning for high-DOF grippers. We advocate the use of Interaction Bisector Surface to characterize the fine-grained spatial relationship between the gripper and the target object. We found that IBS is surprisingly effective as a state/observation representation of deep RL since it well informs the fine-grained control of each finger with spatial relation against the target object. Together with a few critical designs of the learning model and strategy, our method learns high-quality grasping with smooth reaching motion.

Our method has the following limitations:

Our RL model adopts success signal in execution as well as Q1 metric for measuring grasp quality and does not explicitly define the naturalness of a grasp. Therefore, there is the case that a generated grasp is successful but looks unnatural. For example, some successful grasps could have an unbent finger that looks implausible; see Figure 13(a).

The other reward we use is devised to avoid collision during the reach-and-grasp process. This makes our method unable to learn picking up a flat shape lying on the table. See Figure 13(b) for an example.

As future work, we would like to conduct further investigation on the following four aspects:

To further improve the performance of our method, we can try more complicated feature encoding other than the current PointNet and MPLs, and further, take more dynamic information of each frame such as velocity into account.

To generate more natural grasps, it is necessary to investigate grasp quality metrics that can better reflect grasp naturalness. This could be learned from human grasping datasets such as ContactPose (Brahmbhatt et al., 2020) .

To make our method be able to grasp flat-shaped objects, we can relax the collision constraint and even utilize the collision with the environment to help lift the object and achieve a successful grasp as in (Eppner et al., 2015) .

To be able to conduct real robot implementation, we need to study how to perform sim-to-real policy transfer, overcoming the domain gap between simulated observation and real visual perception, as well as the gap between simplified static environment and real dynamic scenes.

ACKNOWLEDGEMENTS

This work was supported in parts by NSFC (61872250, 62132021, U2001206, U21B2023, 62161146005), GD Talent Plan (2019JC05X328), GD Natural Science Foundation (2021B1515020085), DEGP Key Project (2018KZDXM058, 2020SFKC059), National Key R&D Program of China (2018AAA0102200), Shenzhen Science and Technology Program (RCYX20210609103121030, RCJC20200714114435012, JCYJ20210324120213036), and Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ).

References