ContactOpt: Optimizing Contact to Improve Grasps

Patrick Grady, Chengcheng Tang, Christopher D. Twigg, Minh Vo, Samarth Brahmbhatt, Charles C. Kemp

Introduction

The availability of data, hand and body models, and learning algorithms has fueled a growing interest in capturing, understanding, and simulating hand-object interactions . Recent algorithms can predict hand and object pose increasingly accurately from an image. However, inferred poses continue to exhibit sufficient error to cause unrealistic hand-object contact, making downstream tasks in simulation, virtual reality, and other applications challenging.

A key issue is that physical contact is sensitive to small changes in pose. For example, less than a millimeter change in the pose of a fingertip normal to the surface of an object can make the difference between the object being held or dropped on the floor. In addition to physical implausibility, lack of contact and other small-scale phenomena can reduce the perceptual realism of rendered poses.

In this paper we present ContactOpt, an algorithm that improves the quality of hand-object contact by refining hand pose. When given a hand mesh and an object mesh, ContactOpt infers where contact is likely to occur and then optimizes the hand pose to achieve this contact.

As shown in Figure 1, ContactOpt consists of two main components, DeepContact and DiffContact. DeepContact is a network that takes the hand and object meshes as input and estimates regions of likely contact. DiffContact is a differentiable function that takes the hand and object meshes as input and outputs contact based on current geometry. ContactOpt uses gradient-based optimization to find pose, translation, and rotation parameters for the MANO hand model that improve the match between current contact from DiffContact and target contact from DeepContact.

Notably, ContactOpt takes into account soft tissue deformation in the hand. The inner surface of a human hand undergoes significant deformation when making contact with objects. For example, the finger pad can deform 2-3 mm, and the palm can deform 5 mm under normal grasping forces . DiffContact permits up to 2 mm of interpenetration between the hand and object meshes without penalty. In addition, ContactOpt’s gradient-based optimization uses a loss function that only penalizes penetration greater than this threshold. This allows for contact to occur across wide areas of the hand, rather than only at single points.

We conducted two types of evaluations to assess ContactOpt’s performance. For the first type of evaluation, we evaluated ContactOpt’s ability to refine hand pose estimates with small inaccuracies in dataset annotations. This presents methodological challenges due to limits in the precision of dataset ground truth annotations. To overcome this, we used the ContactPose dataset, which has both pose estimates and measured contact data obtained via thermal imagery. We had ContactOpt refine these hand pose estimates with respect to ground truth contact. The refined hand poses better matched ground truth contact and were preferred by human participants, demonstrating that ContactOpt can improve state-of-the-art pose estimates from existing datasets.

For the second type of evaluation, we evaluated ContactOpt’s ability to refine hand pose estimates with large inaccuracies. We used ContactOpt to refine hand pose estimates from an existing RGB hand pose estimation network (Hasson et al. ) applied to the HO-3D dataset . ContactOpt’s refined hand poses had lower kinematic error, were preferred by human participants, and matched more closely to previously observed hand contact patterns (Figure 2). ContactOpt also outperformed RefineNet (an end-to-end grasp refinement network) with respect to both measures. This demonstrates ContactOpt’s value as a post-processing stage for existing hand-object pose estimation algorithms for which it has not been specifically trained. Since ContactOpt operates on hand and object meshes, it has the potential to improve the output of recent image-based estimation methods while avoiding some types of generalization issues associated with operating on images.

We show that methods that explicitly consider hand-object contact can improve hand pose estimates at both coarse ( $\approx$ cm) and fine ( $\approx$ mm) spatial scales, resulting in improved visual realism and lower kinematic error.

We present DeepContact, a deep network that estimates where contact is likely to occur across the surfaces of inaccurately aligned hand and object meshes.

We present DiffContact, a differentiable contact model that estimates where contact is occurring between hand and object meshes.

We present ContactOpt, an algorithm that performs gradient-based optimization to improve hand-object contact by refining hand pose.

Related Work

In this work, we use likely contact and a contact model to improve the pose of a hand grasping an object. Applications in computer vision, animation, and robotics have driven interest in hand-object interaction tracking from different angles, e.g., recovering poses from input images or generating grasps based on object pose and geometry. Information about contact is playing an increasingly important role for hand-object interaction tracking, grasp generation and multiple other related applications.

Recently, there has been a focus on collecting datasets that include interactions between hands and objects. FreiHand uses multiple cameras to extract high-quality annotations using the MANO model, but does not include the object pose. HO-3D optimizes simultaneously for both hand and object poses from RGB-D sensors. FHAB leverages a unique magnetic tracking system to infer the pose of a hand and object even under occlusion. GRAB uses professional optical motion capture to collect a dataset of people grasping and manipulating objects. The work additionally infers contact from the proximity of hand and object. However, these estimates may be noisy due to the very high pose accuracy necessary to infer accurate contact.

Datasets for contact directly measured on objects and hands are complementary to datasets on hand-object poses. The ContactPose dataset is unique in capturing both ground truth thermal contact maps, as well as hand and object pose. The participants held a static grasp for each of 25 objects while being captured using multiple RGB-D cameras. The object was tracked using motion capture, and the hand pose was estimated by aggregating predictions across time from an RGB hand pose estimator. A thermal camera measured the body heat transferred from the participant’s hand to the object, providing ground truth contact. The dataset shows that contact occurs across large sections of the hand, as opposed to only at the fingertips. A limitation of the method is that the 3D hand pose accuracy is bounded by the accuracy of the hand pose estimation, so there may be discrepancies between the contact map and the MANO hand mesh.

There is an extensive body of work on estimating the pose of the hand using a variety of input modalities, including: gloves with markers or sensors , depth/RGB-D input , and RGB or monochrome images , with an increasing focus on hand-object interaction . Researchers have long realized that inferring and enforcing contact is important for hand-object interaction tracking , and it remains a challenging task, particularly in the absence of depth data. For RGB-D hand tracking, hand-object contact modeled as finger-tip to object distance was part of the energy function during optimization with Gaussian Mixture Models in . For image-based prediction, skeletal hand poses or MANO hand model parameters are predicted jointly with object geometry or pose in an end-to-end manner. Despite sharing a joint latent space, since the output representations for the hand and object are decoupled, there can be relative errors in the poses, leading to unrealistic grasps. Even though contact can be encouraged at training time, these networks have no method of enforcing alignment at test time. Our work complements these existing methods by leveraging the strength of their joint hand-object pose prediction, but uses explicit contact inference and enforcement to achieve higher quality grasps.

Robotic grasp generation shares many similarities to pose refinement. Generally, the robot attempts to find a stable grasp with high robustness to perturbations. Various input modalities have been explored for learned grasp detectors, including depth and RGB . Some methods use physics simulation or analytical heuristics to find stable grasps. The majority of robotic grasping work focuses on simple grippers with sparse contact points, however some research has investigated manipulation with anthropomorphic hands .

Similarly, generating plausible grasps for a human hand has also been explored. In GanHand , a dataset of affordances and grasps was proposed to generate plausible human grasps based on input images. The works that are most similar to ours are ContactGrasp and GRAB . In ContactGrasp , dense ground truth contact maps from ContactDB are used to generate plausible grasps for a given object geometry. However, this requires pre-recorded contact maps, and because the ContactDB dataset lacks ground truth hand poses, they cannot compare against ground truth or condition on images as we do. In GRAB the authors leverage their collected data to generate compelling grasps for a variety of objects. They propose RefineNet, which improves the quality of a grasp given an initial pose. This has similarities to our approach, but it performs end-to-end pose updates rather than optimization, and considers fixed contact patterns as opposed to contact estimated separately for each grasp. The method does not explicitly consider object geometry, and because it is fully learned, may have less ability to generalize. We show comparisons against this approach when applied to image-based inference tasks in \secrefsec:experiments.

Aside from hand-object interaction, contact is informative for full human body poses given human-environment interaction . Inferred contact constraints are used in to improve body pose estimation from videos to mitigate artifacts such as feet sliding. Coarse contact points are used in generating human poses interacting with scenes . Our work leveraging fine-grain contact information to improve hand pose in hand-object interaction tracking is related to and likely applicable to context-aware full-body pose estimation and generation.

Methods

We represent the grasp with an object mesh $\mathbf{O}$ and a MANO hand mesh $\mathbf{H}$ . $\mathbf{H}$ is described by parameters $\mathbf{P}=\left(\theta,\beta,{t}^{H},{R}^{H}\right)$ , consisting of pose, shape, translation, and rotation w.r.t. the object respectively. Pose $\theta$ is represented as a 15-dimensional PCA manifold, which lowers the high-dimensional joint angle representation to a compressed space of typical hand poses.

Given a noisy estimate of $\mathbf{P}$ (which typically comes from an image-based algorithm), we seek a better grasp by exploiting the hand-object contact information. Figure 1 shows an overview of our approach. In the following sections, we describe our learned contact map estimation module DeepContact (Section 3.1) and our differentiable contact model DiffContact (Section 3.2) that is iteratively updated according to the optimized hand pose to reproduce the estimated contact (Section 3.3).

Given an object mesh $\mathbf{O}$ and and hand mesh $\mathbf{H}$ with potentially inaccurate pose $\mathbf{P}$ , DeepContact learns to infer target contact on the hand $\hat{C}_{H}$ and object $\hat{C}_{O}$ .

We represent the meshes $\mathbf{H}$ and $\mathbf{O}$ as point clouds, and use PointNet++ to predict contact. The object pointcloud contains 2048 points randomly sampled from the object. The hand point cloud contains all 778 vertices of the MANO mesh. We employ the “mesh” features, training loss, and discrete contact representation of Brahmbhatt et al. . The “mesh” features capture distances from the hand to the object, as well as normal information. Additionally, we include a binary per-point feature indicating whether the point belongs to the hand or the object. The network predicts contact as a classification task, where the range $$ is split into 10 bins. We train DeepContact with the standard binary cross-entropy loss.

Similarly to GrabNet , we train this module on a dataset of randomly perturbed hand poses from the ContactPose dataset, which we call Perturbed ContactPose. The hand mesh is modified by adding noise to the parameters $\Delta\theta\sim\mathcal{N}(0,0.5)$ , $\Delta{t}^{H}\sim\mathcal{N}(0,5)$ cm, and $\Delta{R}^{H}\sim N(<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mo separator="true">,</mo></mrow><annotation encoding="application/x-tex">,</annotation></semantics></math>,)$ . Object contact is supervised with ground-truth thermal contact from ContactPose. To generate the target hand contact map, we run DiffContact (section 3.2). By applying multiple perturbations to each grasp, a training/testing split of 22K / 1.4K grasps is generated.

Figure 3a shows example perturbations, and Figure 3b shows an example contact prediction. Hand and object poses that are farther from a particular grasp tend to result in larger and more diffuse areas of predicted contact.

2 DiffContact: Differentiable Contact Model

DiffContact estimates the contact maps, $C_{O}(\mathbf{P})$ and $C_{H}(\mathbf{P})$ , based on the current meshes $\mathbf{O},\mathbf{H}(\mathbf{P})$ . This is done in a differentiable way, allowing optimization of the hand parameters $\mathbf{P}$ .

We propose a contact model using virtual capsules, as shown in Figure 4a. Our virtual capsules have useful attraction extended beyond the surface (which a binary proximity would not) and approximate soft hand tissue deformation.

More concretely, we place a virtual capsule at every object vertex $\mathbf{v}^{O}_{i}$ and orient it along the object surface normal $\mathbf{n}^{O}_{i}$ . This capsule has a principal line segment defined by $\mathbf{v}^{O}_{i}+\alpha\mathbf{n}^{O}_{i}$ , $\alpha\in[-c_{\text{bot}},c_{\text{top}}]$ . Let $\phi(\mathbf{x})$ be the Euclidean distance from a 3D point $\mathbf{x}$ to this line segment. The contact is defined to be uniformly 1 for points such that $\phi(\mathbf{x})<c_{\text{rad}}{}$ and falls off proportionally with distance outside $c_{\text{rad}}{}$ as $\frac{c_{\text{rad}}}{\phi(\mathbf{x})}$ .

Let $\mathbf{v}_{j}^{H}(\mathbf{P})$ be the hand vertex at pose $\mathbf{P}$ with the smallest distance $\phi$ to the object vertex $\mathbf{v}_{i}^{O}$ . The contact value at the object vertex $\mathbf{v}_{i}^{O}$ is expressed as:

The same procedure can be used to calculate the contact map on the hand surface. We choose an asymmetric $c_{\text{bot}}>c_{\text{top}}$ such that the region considered “in contact” extends farther inside the mesh than outside, which approximates soft hand tissue deformation as shown in Figure 3c. In our experiments, $c_{\text{top}}=0.5$ mm, $c_{\text{bot}}=1$ mm, and $c_{\text{rad}}=1$ mm. As the total capsule depth inside the object is $c_{\text{bot}}{}+c_{\text{rad}}{}=2$ mm, this conservatively matches the $2-3$ mm finger pad deformation found in the biomechanics literature .

Figure 4b shows an example of object contact computed with this model. Because the generated contact has a gradual dropoff, this provides gradients for optimization. Additionally, the resulting contact maps have diffuse edges, which appear visually similar to thermal contact maps . The generated contact is an area instead of a single point.

3 Contact Optimization

To align the meshes, the hand mesh parameters $\mathbf{P}$ are iteratively optimized (Figure 4c) to minimize the difference between the current contact maps $C_{H}(\mathbf{P})$ , $C_{O}(\mathbf{P})$ computed using DiffContact, and the target contact maps $\hat{C}_{H}$ , $\hat{C}_{O}$ as predicted by DeepContact, or from ground truth thermal contact.

The contact loss for the object surface is:

Here we use $\lambda>1$ to penalize “missing” contacts (where the target contact is higher than the value estimated by DiffContact) more heavily than “unexpected” contacts. This is based on the empirical observation that it is visually worse for the hand to “hover” over the object than to be slightly interpenetrating. We apply a corresponding loss $E_{H}(\mathbf{P})$ to penalize differences between the target hand contact map $\hat{C}_{H}$ and $C_{H}(\mathbf{P})$ . We use $\lambda=3$ in both cases.

We also include an explicit penetration term that penalizes penetrations beyond $c_{pen}$ . This discourages heavy intersection where vertices on the back of the hand register as in contact. For each object vertex $\mathbf{v}^{O}_{i}$ , object surface normal $\mathbf{n}^{O}_{i}$ , and nearest hand vertex $\mathbf{v}^{H}_{j}(\mathbf{P})$ , the penetration loss is defined as

The loss is minimized by the ADAM optimizer using gradients computed with PyTorch automatic differentiation . We use a learning rate of $0.01$ and optimize for 250 iterations. Optimizing a batch of 64 hand-object pairs takes 4 s (amortized runtime 62 ms). We scale the gradients for the different components of $\mathbf{P}$ . See the supplementary material for more details.

Since the contact optimization is local, a poor initialization (\eginitial hand position on the wrong side of an object) can result in the optimizer settling into a bad local minimum. We avoid this by applying the pose optimization to several perturbations of the provided pose and select the result with the lowest loss.

Evaluation

We evaluate how well ContactOpt improves poses with small inaccuracies and with large inaccuracies using the ContactPose and HO-3D datasets. In each case, the refined hand mesh is evaluated using the following metrics.

Intersection Volume (cm3): Intersection volume of $\mathbf{H}$ and $\mathbf{O}$ , calculated from their mesh intersection. Standard deviation across the dataset is also shown.

Mean Per-Joint Position Error (MPJPE) (mm): Average L2 per-joint kinematic error with respect to the ground truth hand .

Contact Coverage (%): Percentage of hand points between -2 mm and +2 mm of the object surface (i.e., approximately in contact with the object).

Contact Precision/Recall (%): Quantifies how well the contact from the refined hand mesh matches the thermal contact map. A binary object contact map is obtained by considering the object points within $\pm$ 2 mm of the hand surface to be in contact. Precision and recall are calculated by comparing this to the thermal contact map thresholded at 0.4, following .

Perceptual Evaluation (%): Nine evaluators who were unfamiliar with the research were recruited to judge the relative quality of grasps in two-alternative forced choice tests (2AFC). Each participant was shown two hand-object pairs and asked to judge “Which looks more like the way a person would grasp the object?”. In pilot studies, we found that non-experts had difficulty comparing grasps with small differences, so pairs with less than a 5 mm MPJPE difference were removed. For each method, the evaluators judged 75 pairs of grasps with an equal number randomly selected for each object. The mean and 95% confidence intervals are shown. More details of this evaluation can be found in the supplementary material.

We use the ContactPose dataset to evaluate the ability of ContactOpt to improve poses with small inaccuracies. Recent hand-object interaction datasets use a variety of techniques to capture hand and object pose, such as magnetic trackers, multi-view reconstruction from RGB-D cameras, or motion capture systems. Despite using high quality sensors, errors on the centimeter-level are not uncommon (Figure 5).

However, when considering the realism of grasps, millimeters matter. Gaps between the hand and object result in unstable grasps and can be visually unsatisfying. Similarly, unrealistic penetration can violate basic assumptions of intact hands and objects. Notably, millimeters of Euclidean error can result in a physically implausible grasp.

ContactOpt can be used to resolve these types of errors when applied to already high-quality poses provided by dataset annotations.

Refining ContactPose Dataset Poses: Millimeter-scale refinement is demonstrated by refining the ContactPose annotated hand meshes. Rather than estimating target contact using DeepContact, the ground truth thermal contact map is used. As ground truth hand contact is not available, hand contact is not used. Table 1 and Figure 7 show the results of this experiment.

Both contact recall and precision metrics increase, demonstrating that ContactOpt improves the self-consistency between ground truth contact and mesh poses. Both unwanted contact as well as excess contact are reduced (Figure 7).

However, it is difficult to quantify the holistic quality of a grasp. We perform a perceptual evaluation where human participants choose the most natural-looking grasp. Contact maps are not shown to the participants. As shown in Table 1, participants favored the refined grasps at over a 2:1 ratio. ContactOpt is able to consistently resolve cases of millimetric penetration or under-shoot and pull the fingers into realistic contact with the object, which is likely noticed by the participants.

This demonstrates that contact and accurate poses can be used together to achieve higher quality than is possible with pose alone.

2 Refining Large Inaccuracies

We evaluate the ability of ContactOpt to improve poses with large inaccuracies in two ways. First, we use perturbed poses from the ContactPose dataset. Second, we use poses estimated from images.

We test the full ContactOpt pipeline on Perturbed ContactPose (Section 3.1), which contains poses with an MPJPE of $\sim$ 80 mm. This tests the ability to improve hand poses with large errors. Results are shown in Figure 8 and Table 1.

Despite being initialized from a heavily misaligned hand pose, the pipeline is still able to reduce kinematic error (MPJPE) by almost 70% and improves perceptual grasp quality. Additionally, the refined meshes are more consistent with the ground truth contact maps, even though they are not provided to the algorithm.

However, some kinematic error remains. Qualitatively, this is because the objects have many valid grasp modes (\iegrasping an apple in any rotation), and it is not possible to recover the correct one from the inaccurate initial pose. Although most refined meshes are visually high quality, often a slight translation results in a large kinematic error.

2.2 Refining Image-Based Pose Estimates

We evaluate ContactOpt in refining the predictions from an image-based pose estimator. In this task, 3D hand and object pose are often estimated using CNNs. For approaches that operate on single-frame RGB images, errors in the multiple-centimeter range are typical, leading to physically implausible grasps. Note that in this setting, there are no image-based constraints placed on the optimization, thus allowing greater freedom of pose refinement.

We use the baseline pose estimation network from Hasson et al. (2020) and retrain it on a training split of the HO-3D dataset. As the network’s object predictions are often unstable, the object class and pose are taken from ground truth. Additionally, poses where the ground truth is not in contact are filtered out. More details can be found in the supplementary material.

We demonstrate that DeepContact is able to generalize well to new datasets. Despite being trained on the Perturbed ContactPose dataset, it can still improve estimates on HO-3D, which has both different objects and features dynamic grasps. Generally, since hand and object geometry is mostly consistent across datasets, the domain gap is smaller than modalities such as RGB, where learned methods often must be completely retrained. We qualitatively find that DeepContact is able to transfer hand contact more reliably than object contact, as the hand representation (MANO) is consistent across datasets.

Results from this task are found in Table 2. Human evaluators favored the refined grasps over the initial grasp estimates by a ratio of almost 6:1. Additionally, the frequency of contact across the hand for the refined grasps (Figure 2) is similar to ground truth frequencies of contact, while the frequency of contact for originally inferred grasps does not resemble normal grasping patterns.

As the dataset contains shapes with many grasp modes (\ieboxes may be grasped anywhere along the edge), DeepContact has difficulty predicting the correct grasp location from a low quality inferred grasp. Figure 8 shows a refined grasp with high perceptual quality but a large MPJPE error metric. Despite this, ContactOpt is still able to lower the mean kinematic joint error by 20%.

Comparing to Baseline Refinement: We also compare ContactOpt to a baseline hand pose refinement method. RefineNet is an end-to-end model trained on the GRAB dataset to refine initial coarse grasp proposals. Given a hand and object mesh, the network predicts pose, rotation, and translation updates. As RefineNet is an iterative method, it is benchmarked with 3 iterations (following the paper) and 10 iterations.

Ablating Random Restarts: The effect of random restarts on kinematic error is shown in Table 3. Due to the non-convexity of the optimization objective, performing random initializations with perturbations to translation improves the performance of ContactOpt.

Conclusion

We introduce ContactOpt, a method to refine coarsely aligned hand and object meshes. DeepContact estimates likely contact on both the hand and the object. DiffContact then estimates contact based on the current mesh pose. The error between these two estimates is used to optimize hand pose to achieve the target contact.

We show that ContactOpt is able to improve both dataset-quality meshes when ground truth thermal contact is provided, as well as pose estimations from images, even when tested on a novel object set. In our experiments, optimized grasps achieved lower kinematic error and were preferred by human evaluators.

Acknowledgements: We thank the anonymous reviewers for their comments to improve this work. We also thank Robert Wang, Yuting Ye, Shangchen Han, Beibei Liu, Chengde Wan, Jeff Petkau, and Henry Clever for their advice and discussions.