ContactPose: A Dataset of Grasps with Object Contact and Hand Pose

Introduction

A person’s daily experience includes numerous and varied hand-object interactions. Understanding and reconstructing hand-object interaction has received growing attention from the computer vision, computer graphics, and robotics communities. Most research has focused on hand pose estimation , realistic hand and body reconstruction , and robotic grasp prediction for anthropomorphic hands . In this paper, we address the under-explored problem of hand-object contact modeling i.e. predicting object contact with the hand, based on other information about the grasp, such as the 3D hand pose and grasp images. Accurate contact models have numerous applications in computer interfaces, understanding social interaction, object manipulation, and safety. For example, a hand contact model could interpret computer commands from physical interactions with a 3D printed replica object, or estimate if pathogens from a contaminated surface were transmitted through contact. More broadly, accurate contact modeling can improve estimation of grasp dynamics , which can lead to better VR simulations of grasping scenarios and grasping with soft robotic hands .

Lack of ground-truth data has likely played a role in the under-exploration of this problem. Typically, the contacting surfaces of a grasp are occluded from direct observation with visible light imaging. Approaches that instrument the hand with gloves can subtly influence natural grasping behavior, and do not measure contact on the object surface. Approaches that intersect hand models with object models require careful selection of proximity thresholds or specific contact hand points . They also cannot account for soft hand tissue deformation, since existing state-of-the-art hand models are rigid.

Brahmbhatt et al. recently introduced thermal cameras as sensors for capturing detailed ground-truth contact. Their method observes the heat transferred from the (warm) hand to the object through a thermal camera after the grasp. We adopt their method because it avoids the pitfalls mentioned above and allows for evaluation of contact modeling approaches with ground-truth data. However, it also imposes some constraints. 1) Objects have a plain visual texture since they are 3D printed to ensure consistent thermal properties. This does not affect 3D hand pose-based contact modeling methods and VR/robotic grasping simulators, since they rely on 3D shape and not texture. It does limit the generalization ability of RGB-based methods, which can potentially be mitigated by using depth images and synthetic textures. 2) The grasps are static, because in-hand manipulation results in multiple overlapping thermal hand-prints that depend on timing and other factors. Contact modeling for static grasps is still an unsolved problem, and forms the basis for future work on dynamic grasps. The methods we present here could be applied to dynamic scenarios frame-by-frame.

In addition, we develop a data collection protocol that captures multi-view RGB-D videos of the grasp, and an algorithm for 3D reconstruction of hand joints (§ 3.1). To summarize, we make the following contributions:

Data: Our dataset (ContactPose) captures 50 participants grasping 25 objects with 2 functional intents. It includes high-quality contact maps for each grasp, over 2.9 M RGB-D images from 3 viewpoints, and object pose and 3D hand joints for each frame. We will make it publicly available to encourage research in hand-object interaction and pose estimation.

Analysis: We dissect this data in various ways to explore the interesting relationship between contact and hand pose. This reveals some surprising patterns, and confirms some common intuitions.

Algorithms: We explore various representations of object shape, hand pose, contact, and network architectures for learning-based contact modeling. Importantly, we rigorously evaluate these methods (and heuristic methods from the literature) against ground-truth unique to ContactPose.

Related Work

Capturing and modeling contact: Previous works have instrumented hands and/or objects to capture contact. Bernardin et al. and Sundaram et al. used a tactile glove to capture hand contact during grasping. Brahmbhatt et al. used a thermal camera after the grasp to observe the heat residue left by the warm hand on the object surface. However, these datasets lacked either hand pose or grasp images, which are necessary for developing applicable contact models (Figure 2). Pham et al. and Ehsani et al. tracked hands and objects in videos, and trained models to predict contact forces and locations at fingertips that explain observed object motion. In contrast, we focus on detailed contact modeling for complex objects and grasps, evaluated against contact maps over the entire object surface.

Contact heuristics: Heuristic methods to detect hand-object contact are often aimed at improving hand pose estimation. Hamer et al. performed joint hand tracking and object reconstruction , and inferred contact only at fingertips using proximity threshold. In simulation and robotic grasping , contact is often determined similarly, or through collision detection . Ballan et al. defined a cone circumscribing object mesh triangles, and penalized penetrating hand points (and vice versa). This formulation has also been used to penalize self-penetration and environment collision . While such methods were evaluated only through proxy tasks (e.g. hand pose estimation), ContactPose enables evaluation against ground-truth contact (§ 6).

Grasp Datasets: Focusing on datasets involving hand-object interaction, hand pose has been captured in 3D with magnetic trackers , gloves , optimization , multi-view boot-strapping , semi-automated human-in-the-loop , manually , synthetically , or as instances of a taxonomy along with RGB-D images depicting the grasps. However, none of these have contact annotations (see Table 1), and suffer additional drawbacks like lack of object information and simplistic objects and interactions , which make them unsuitable for our task. In contrast, ContactPose has a large amount of ground-truth contact, and real RGB-D images of complex (including bi-manual) functional grasps for complex objects. The plain object texture is a drawback of ContactPose. Tradeoffs for this in the context of contact modeling are discussed in § 1.

The ContactPose Dataset

In ContactPose, hand-object contact is represented as a contact map on the object mesh surface, and observed through a thermal camera. Hand pose is represented as 3D hand(s) joint locations in the object frame, and observed through multi-view RGB-D video clips. The cameras are calibrated and object pose is known, so that the 3D joints can be projected into images (examples shown in supplementary material). Importantly, we avoid instrumenting the hands with data gloves, magnetic trackers or other sensors. This has the dual advantage of not interfering with natural grasping behavior and allowing us to use the thermal camera-based contact capture method from . We develop a computational approach (Section 3.2) that optimizes for the 3D joint locations by leveraging accurate object tracking and aggregating over multi-view and temporal information. Our data collection protocol, described below, facilitates this approach.

We invite able-bodied participants to our laboratory and collect data through the following IRB-approved protocol. Objects are placed at random locations on a table in orientation normally encountered in practice. Participants are instructed to grasp an object with one of two functional intents (either using the object, or handing it off). Next, they stand in the data collection area (Figure 3(a)) and move the object for 10-15 s in the cubical space. They are instructed to hold their hand joints steady, but are free to arbitrarily rotate the wrist and elbow, and to grasp objects with both hands or their dominant hand. This motion is recorded by 3 Kinect v2 RGB-D cameras (used for hand pose) and an Optitrack motion capture (mocap) system (used for object pose). Next, they hand the object to a researcher, who places it on a turntable, handling it with gloved hands. The object is recorded with the mocap system, Kinect v2, and a FLIR Boson 640 thermal camera as the turntable rotates a circle.

Contact Capture: Thermal images are texture-mapped to the object mesh using Open3D . As shown in and the supp. mat., the resulting mesh textures (called contact maps) accurately capture hand-object contact.

Object Selection and Fabrication: We capture grasps on a subset of 25 objects from that are applicable for both ‘use’ and ‘hand-off’ grasping (see supp. mat. for a list). The objects are 3D printed in blue for good contrast with hands and the green background of our capture area. 3D printing the objects ensures consistent thermal properties and ensures geometric consistency between real world objects in capture sessions and the 3D models in our dataset.

Mocap recovers the object pose using retro-reflective markers, whose the placement on the object requires some care. Attaching a large ‘marker tree’ would block interactions with a significant area of the surface. Placing hemispherical markers on the surface is more promising, but a sufficient number (8+) are needed to ensure visibility during hand occlusion and the resulting ‘bumps’ can be uncomfortable to touch, which might influence natural grasping behavior. We investigate a few alternative marker configurations (Figure 3(b)). Flat pieces of tape were more comfortable but only tracked well when the marker was directly facing the camera. A good compromise is to use 3 mm hemispherical markers but to recess them into the surface by adding small cut-outs during 3D printing. These are visible from a wide range of angles but do not significantly affect the user’s grip. Fixing the marker locations also allows for simple calibration between the Optitrack rigid body and the object’s frame.

2 Grasp Capture without Hand Markers

Each grasp is observed through $N$ frames of RGB-D images from $C$ cameras. We assume that the hand is fixed relative to the object, and the 6-DOF object pose for each frame is given. So instead of estimating 3D joints separately in each frame, we can aggregate the noisy per-frame 2D joint detections into a single set of high-quality 3D joints, which can be transformed by the frame’s object pose.

For each RGB frame, we use Detectron to locate the wrist, and run the OpenPose hand keypoint detector on a 200$\times$200 crop around the wrist. This produces 2D joint detections $\{\mathbf{x}^{(i)}\}_{i=1}^{N}$ and confidence values $\{\mathbf{w}^{(i)}\}_{i=1}^{N}$, following the 21-joint format from . One option is to lift these 2D joint locations to 3D using the depth image , but that biases the location toward the camera and the hand surface (our goal is to estimate joint locations internal to the hand). Furthermore, the joint detections at any given frame are unreliable. Instead, we use our hand-object rigidity assumption to estimate the 3D joint locations $\mathbf{{}^{o}X}$ in the object frame that are consistent with all $NC$ images. This is done by minimizing the average re-projection error:

where $\mathcal{D}$ is a distance function, and $\pi(\cdot)$ is the camera projection function using camera intrinsics $K_{c}$ and object pose w.r.t. camera at frame $i$, ${}^{c}T_{o}^{(i)}=~{}^{c}T_{w}~{}^{w}T_{o}^{(i)}$. Our approach requires the object pose w.r.t. world at each frame ${}^{w}T_{o}^{(i)}$ i.e. object tracking. This is done using an Optitrack motion capture system tracking markers embedded in the object surface.

In practice, the 2D joint detections are noisy and object tracking fails in some frames. We mitigate this by using the robust Huber function over Mahalanobis distance ($\mathbf{w}^{(i)}$ acting as variance) as $\mathcal{D}$, and wrapping Eq. 1 in a RANSAC loop. A second pass targets frames that fail the RANSAC inlier test due to inaccurate object pose. Their object pose is estimated through the correspondence between their 2D detections and the RANSAC-fit 3D joint locations, and they are included in the inlier set if they pass the inlier test (re-projection error less than a threshold). It is straightforward to extend the optimization described above to bi-manual grasps. We manually curated the dataset, including clicking 2D joint locations to aid the 3D reconstruction in some cases, and discarding some obviously noisy data.

Hand Mesh Models: In addition to capturing grasps, hand shape information is collected through palm contact maps on a flat plate, and multi-view RGB-D videos of the participant performing 7 known hand gestures (shown in the supplementary material). Along with 3D joints, this data can potentially enable fitting of the MANO hand mesh model to each grasp . In this paper, we use meshes fit to 3D joints (Figure 4, see supp. mat. for details) for some of the analysis and learning experiments discussed below.

Data Analysis

Contact maps are $$ normalized following the sigmoid fitting procedure from .

Association of Contact to Hand Parts: It has been observed that certain fingers and parts (e.g. fingertips) are contacted more frequently than others . ContactPose allows us to quantify this. This can potentially inform anthropomorphic robotic hand design and tactile sensor (e.g. BioTac ) placement in robotic hands. For each grasp, we threshold the contact map at 0.4 and associate each contacted object point with its nearest hand point from the fitted MANO hand mesh. A hand point is considered to be contacted if one or more contacted object points are associated with it. A coarser analysis at the phalange level is possible by modeling phalanges as line segments connecting joints. In this case, the distance from an object point to a phalange is the distance to the closest point on the line segment.

Figure 5(a) shows the contact probabilities averaged over ‘use’ and ‘hand-off’ grasps. Not surprisingly, the thumb, index, and middle finger are the most contacted fingers, and tips are the most contacted phalanges. Even though fingertips receive much attention in grasping literature, the contact probability for all three phalanges of the index finger is higher than that of the pinky fingertip. Proximal phalanges and palm also have significant contact probabilities. This is consistent with observations made by Brahmbhatt et al . Interestingly, contact is more concentrated at the thumb and index finger for ‘hand-off’ than ‘use’. ‘Use’ grasps have an average contact area of 35.87 cm2 compared to 30.58 cm2 for ‘hand-off’. This analysis is similar to that in Fig. 3 of Hasson et al., but supported by ground-truth contact rather than synthetic grasps.

Comparison of the average fingertip vs. whole-hand contact areas (Figure 6) shows that non-fingertip areas play a significant role in grasp contact, confirming the approximate analysis in .

Automatic Active Area Discovery: Brahmbhatt et al define active areas as regions on the object highly likely to be contacted. While they manually selected active areas and measured their probability of being contacted by any part of the hand, ContactPose allows us to ‘discover’ active areas automatically and for specific hand parts. We use the object point-phalange association described above (e.g. Fig. 5(b)) to estimate the probability of each object point being contacted by a given hand part (e.g. index finger tip), which can be thresholded to segment the active areas. Figure 7 shows this probability for the index fingertip and thumb, for ‘use’ grasps of some objects. This could potentially inform locations for placing contact sensors (real or virtual for VR) on objects.

Grasp Diversity: We further quantify the effect of intent on grasping behavior by measuring the standard deviation of 3D joint locations over the dataset. The mean of all 21 joint standard deviations is shown in Figure 8(a). It shows that ‘hand-off’ grasps are more diverse than ‘use’ grasps in terms of hand pose. We accounted for symmetrical objects (e.g. wine glass) by aligning the 6 palm joints (wrist + 5 knuckles) of all hand poses for that object to a single set of palm joints, where the only degree of freedom for alignment is rotation around the symmetry axis. Hand size is normalized by scaling all joint location such that the distance from wrist to middle knuckle is constant.

Organizing the grasps by clustering these aligned 3D joints (using L2 distance and HDBSCAN ) reveals the diversity of grasps captured in ContactPose (Figure 9). ‘Hand-off’ grasps exhibit a more continuous variation than ‘use’ grasps, which are tied more closely to the function of the object. The average intra-cluster distance for ‘use’ grasps is 32.5% less than that for ‘handoff’ grasps.

Figure 8(b) shows pair of grasps found by minimizing hand pose distance and maximizing hand contact distance. We use the phalange-level contact association described above. Summing the areas of all object mesh triangles incident to all vertices associated with a phalange creates a 20-dimensional vector. We use L2 distance over this vector as contact distance. It shows that grasps with similar hand pose can contact different parts of the object and/or hand, inducing different forces and manipulation possibilities and emphasizing that hand pose alone provides an inadequate representation of grasping.

Contact Modeling Experiments

This section describes our experiments on contact modeling given the hand pose or RGB grasp image(s), assuming known object geometry and pose. Our experiments focus on finding good data representations and learning algorithms, and evaluating techniques against ground-truth. By providing high-quality contact output from readily available input modalities, such models can enable better hand-object dynamics simulation in AR/VR and soft robotic grasping.

Object Shape Representation: We represent the object shape through either a pointcloud densely sampled from the surface (1K-30K points based on size), or a $64^{3}$ voxel occupancy grid. Features encoding the input hand pose are associated with individual points (voxels). The entire pointcloud (voxel grid) is then processed to predict contact values for points (surface voxels).

Hand Pose Representation: Features relating object shape to hand pose are computed for each point or voxel. These features have varying levels of richness of hand shape encoding. To simulate occlusion and noisy pose perception for the first 4 features, we sample a random camera pose and drop (set to 0) all features associated with the farthest 15% of the joints from the camera.

simple-joints: We start by simply using the 21 3D joint locations w.r.t. the object coordinate system as 63-dimensional features for every point. For bi-manual grasps, points use the hand with the closest joint.

relative-joints: Since contact at an object surface point depends on the relative position of the finger, we next calculate relative vectors from an object point to every joint of the hand closest to it. Contact also depends on the surface geometry: a finger is more likely to contact an object point if the vector to it is parallel to the surface normal at that point. Hence we use unit-norm surface normals and the relative joint vectors to form $63+3=66$-dimensional features for every point.

skeleton: To better capture hand joint connectivity, we compute relative vectors from an object point to the nearest point on phalanges, modeled as line segments. 40-dimensional features for each object point are constructed by concatenating the lengths of 20 such vectors (one for each phalange), and their dot product with the surface normal at that object point.

mesh: These features leverage the rich MANO hand model geometry. A relative vector is constructed from the object point to its closest hand mesh point. 23-dimensional features are constructed from the length of this vector, its dot product with the surface normal, and distances to 21 hand joints.

Grasp Image(s): To investigate if CNNs can extract relevant information directly from images, we extract dense 40-dimensional features from 256$\times$256 crops of RGB grasp images using a CNN encoder-decoder inspired by U-Net (see supplementary material for architecture). These images come from the same time instant. We investigate both 3-view and 1-view settings, with feature extractor being shared across views for the former. Features are transferred to corresponding 3D object points using the known object pose and camera intrinsics, averaging the features if multiple images observe the same 3D point (Figure 11(a)). Points not visible from any image have all features set to 0. Image backgrounds are segmented by depth thresholding at the 20th percentile, and the foreground pixels are composited onto a random COCO image. This investigation is complementary to recent work on image-based estimation of object geometry , object pose , and hand pose .

Contact Representation: We observed in early experiments that the mean squared error loss resulted in blurred and saturated contact predictions. This might be due to contact value occurrence imbalance and discontinuous contact boundaries for smooth input features. Hence, we discretize the $$normalized values into 10 equal bins and treat contact prediction as a classification problem, inspired by Zhang et al . We use the weighted cross entropy loss, where the weight for each bin is proportional to a linear combination of the inverse occurrence frequency of that bin and a uniform distribution (Eq. 4 from with$\lambda=0.4$). Following , we derive a point estimate for contact in$$from classification outputs using the annealed mean ($T=0.1$).

Learning Algorithms: Given the hand pose features associated with points or voxels, the entire pointcloud or voxel grid is processed by a neural network to predict the contact map. We use the PointNet++ architecture implemented in pytorch-geometric (modified to reduce the number of learnable parameters) for pointclouds, and the VoxNet -inspired 3D CNN architecture from for voxel grids (see the supplementary material for architectures). For voxel grids, a binary feature indicating voxel occupancy is appended to hand pose features. Following , hand pose features are set to 0 for voxels inside the object. Because the features are rich and provide fairly direct evidence of contact, we include a simple learner baseline of a multi-layer perceptron (MLP) with 90 hidden nodes, parametric ReLU and batchnorm .

Contact Modeling Heuristics: We also investigate the effectiveness of heuristic techniques, given detailed hand geometry through the MANO hand mesh. Specifically, we use the conic distance field $\Psi$ from as a proxy for contact intensity. To account for imperfections in hand modelling (due to rigidity of the MANO mesh) and fitting, we compute $\Psi$ not only for collisions, but also when the hand and object meshes are closer than 1 cm. Finally, we calibrate $\Psi$ to our ground truth contact through least-squares linear regression on 4700 randomly sampled contact points. Both these steps improve the technique’s performance.

Results

In this section, we evaluate various combinations of features and learning algorithms described in § 5. The metric for quantitative evaluation is the area under the curve formed by calculating accuracy at increasing contact difference thresholds. Following , this value is re-balanced to account for varying occurrence frequencies of values in the 10 contact bins. We create two data splits: the object split holds out mug, pan and wine glass following , and the participant split holds out participants 5, 15, 25, 35, and 45. The held out data is used for evaluation, and models are trained on the rest.

Table 2 shows the re-balanced AuC values averaged over held out data for the two splits. We observe that features capturing richer hand shape information perform better (e.g. simple-joints vs. skeleton and mesh). Learning-based techniques with mesh features that operate on pointclouds are able to outperform heuristics, even though the latter has access to the full high-resolution object mesh, while the former makes predictions on a pointcloud. Learning also enables skeleton features, which have access to only the 3D joint locations, to perform competitively against mesh-based heuristics and features. While image-based techniques are not yet as accurate as the hand pose-based ones, a significant boost is achieved with multi-view inputs.

Figure 10 shows contact prediction results from hand pose for mug, an unseen object. Predictions are transferred from the pointcloud to high-resolution meshes for better visualization. The skeleton-PointNet++ combination is able to predict plausible contact patterns for dropped-out parts of the hand, and capture some of the nuances of palm contact. The mesh-PointNet++ combination captures more nuances, especially at the thumb and bottom of the palm. In contrast, relative-joints features-based predictions are diffused, lack finer details, and have high contact probability in the gaps between fingers, possibly due to lack of access to information about joint connectivity and hand shape.

Figure 11(b) shows contact prediction results from RGB images for mug, an unseen object. These predictions have less high-frequency details compared to hand pose based predictions. They also suffer from depth ambiguity – the proximal part of the index finger appears to be in contact from the mug images, but is actually not. This can potentially be mitigated by use of depth images.

Conclusion and Future Work

We introduced ContactPose, the first dataset of paired hand-object contact, hand pose, object pose, and RGB-D images for functional grasping. Data analysis revealed some surprising patterns, like higher concentration of hand contact at the first three fingers for ‘hand-off’ vs. ‘use’ grasps. We also showed how learning-based techniques for geometry-based contact modeling can capture nuanced details missed by heuristic methods.

Using this contact ground-truth to develop more realistic, deformable hand mesh models could be an interesting research direction. State-of-the-art models (e.g. ) are rigid, while the human hand is covered with soft tissue. As the Future Work section of notes, they are trained with meshes from which objects are manually removed, and do not explicitly reason about hand-object contact. ContactPose data can potentially help in the development and evaluation of hand mesh deformation algorithms.

Acknowledgements: We are thankful to the anonymous reviewers for helping improve this paper. We would also like to thank Elise Campbell, Braden Copple, David Dimond, Vivian Lo, Jeremy Schichtel, Steve Olsen, Lingling Tao, Sue Tunstall, Robert Wang, Ed Wei, and Yuting Ye for discussions and logistics help.

Supplementary Material

Appendix 0.A Contact Capture Discussion

The process to convert thermal image pixels to contact values follows . Raw thermal readings were converted to continuous contact values in $$using a sigmoid that maps the warmest point to$0.95$and the coldest point to$0.05$. These values non-linearly encode the temperature of the object, where$$approximately corresponds to [room temperature, body temperature]. While most experiments used this continuous value, if a hard decision about the contact status of a point was desired, this was done by thresholding these processed values at$0.4$.

Appendix 0.B MANO Fitting

This section provides details for the fitting procedure of the MANO hand model to ContactPose data. Borrowing notation from , the MANO model is a mesh with vertices $M\left(\beta,\theta\right)$ parameterized by shape parameters $\beta$ and pose parameters $\theta$. The 3D joint locations of the posed mesh, denoted here by $J\left(\beta,\theta\right)$, are also a function of the shape and pose parameters. We modify the original model by adding one joint at each fingertip, thus matching the format of joints $J^{*}$ in ContactPose annotations.

MANO fitting is performed by optimizing the following objective function, which combines L2 distance of 3D joints and shape parameter regularization:

where $\sigma$ is set to 10. It is optimized using the Dogleg optimizer implemented in chumpy . We initialized $\beta$ and $\theta$ to $\mathbf{0}$ (mean shape and pose) after 6-DOF alignment of the wrist and 5 palm joints. Finally, the MANO model includes a PCA decomposition of 45 pose parameters to 6 parameters by default. We provide MANO fitting data with 10 and 15 pose components in the ContactPose dataset, but use the MANO models with 10 pose components in all our contact modeling experiments.

Appendix 0.C Dataset Accuracy

In this section, we cross-evaluate the accuracy of the hand pose and contact data included in ContactPose.

We note that conduction is the principal mode of heat transfer in solid-to-solid contact . Combined with the observation by Brahmbhatt et al. that heat dissipation within the 3D printed objects is low over the time scales we employ to scan them, this indicates that conducted heat can accurately encode contact. Following , we measure the conducted heat with a thermal camera.

Agreement with MANO Hand Mesh: The average distance of contacted object points from their nearest hand point is 4.17 mm (10 MANO hand pose parameters) and 4.06 mm (15 MANO hand pose parameters).

Agreement with Pressure-based Contact: We also verified thermal contact maps against pressure images from a Sensel Morph planar pressure sensor . After registering the thermal and pressure images, we thresholded the processed thermal image at values in $$ with an interval of 0.1. Any nonzero pixel in the pressure image is considered to be contacted. Binary contact agreement peaks at 95.4% at the threshold of 0.4 (Figure 12).

C.2 3D Hand Pose Accuracy

Following , this is measured through the discrepancy between 3D joints of the fitted MANO model and the ground truth 3D joints. Low-quality physically implausible ground truth can yield higher discrepancy, since the MANO model is not able to fit to physically implausible joint locations. Table 3 shows that ContactPose has significantly lower discrepancy than other recent datasets, even though it uses less than one-third MANO hand pose parameters. Table 4 shows statistics for hand-object penetration.

Appendix 0.D Participants’ Hand Information

We captured information about each ContactPose participant’s hands in two ways: 1) contact map on a flat plate (example shown in Figure 13), and 2) RGB-D videos of the participants performing 7 hand gestures (shown in Figure 14). This can potentially be used to estimate the hand shape by fit embodied hand models (e.g. ).

Appendix 0.E Network Architectures

The PointNet++ architecture we use is similar to the pointcloud segmentation network from Qi et al , with modifications aiming to reduce the number of learnable parameters. Similarly to , we use $SA\left(s,r,[l_{1},\ldots,l_{d}]\right)$ to indicate a Set Abstraction layer with a farthest point sampling ratio $s$, ball radius $r$ (the pointcloud is normalized to lie in the $[-0.5,0.5]$ cube) and $d$ fully connected layers of size $l_{i}(i=1\ldots d)$. The global Set Abstraction layer is denoted without farthest point sampling ratio and ball radius. $FP\left(K,[l_{1},\ldots,l_{d}]\right)$ indicates a Feature Propagation layer with $K$ nearest neighbors and $d$ fully connected layers of size $l_{i}(i=1\ldots d)$. $FC\left(S_{in},S_{out}\right)$ indicates a fully connected layer of output size $S_{out}$ applied separately to each point (which has $S_{in}$-dimensional features). Each fully connected layer in the Set Abstraction and Feature Propagation layers is followed by ReLU and batch-norm layers. Our network architecture is:

where $F$ is the number of input features and the final layer outputs scores for the 10 contact value classes.

E.2 Image Encoder-Decoder

We take inspiration from U-Net and design the light-weight network shown in 15 that extracts dense features from RGB images. The global average pooling layer is intended to capture information about the entire hand and object.

Appendix 0.F Training and Evaluation Details

All models are trained using PyTorch and the Adam optimizer (base learning rate $\in\{5\times 10^{-4},1\times 10^{-3},5\times 10^{-3}\}$, momentum of 0.9, weight decay of $5e-4$, and a batch size of 25). Both point-clouds and voxel-grids are rotated around their ‘up’-axis at regularly spaced $30\degree$ intervals. These rotations are considered separate data points during training, and their predictions are averaged during evaluation.

For image-based contact prediction, ContactPose has approximately 300 RGB-D frames ($\times$ 3 Kinects) for each grasp, but temporally nearby frames are highly correlated because of the high frame rate. Hence, we include equally spaced 50 frames from each grasp in the training set. Evaluation is performed over equally spaced 12 frames from this set of 50 frames.

Appendix 0.G List of Objects

Table 5 shows a list of all 25 objects in ContactPose, along with information about the which of these objects are included in the two functional grasping categories, and the specific ‘use’ instructions.

References