Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects

Jeffrey Ichnowski, Yahav Avigal, Justin Kerr, Ken Goldberg

Introduction

Transparent objects are common in homes, restaurants, retail packaging, labs, gift shops, hospitals, and industrial warehouses. Effectively automating robotic manipulation of transparent objects could have a broad impact, from helping in everyday tasks and performing tasks in hazardous environments. Existing depth cameras assume that surfaces of observed objects reflect light uniformly in all directions, but this assumption does not hold for transparent objects as their appearance varies significantly under different view directions and illumination conditions due to reflection and refraction. In this paper, we propose and demonstrate Dex-NeRF, a new method to sense the geometry of transparent objects and allow for robots to interact with them—potentially enabling automation of new tasks.

Dex-NeRF uses a Neural Radiance Fields (NeRF) as part of a pipeline (Fig. 1, right) to compute and execute robots grasps on transparent objects. While NeRF was originally proposed as an alternative for explicit volumetric representations to render novel views of complex scenes , it can also reconstruct the geometry of the scene. In particular, due to the view-dependent nature of the NeRF model, it can learn to accurately represent the geometry associated with transparency. The only input requirement to train a NeRF model is a set of images taken from a camera with known intrinsics (e.g., focal length, distortion) and extrinsics (position and orientation in the world). While the intrinsics can be determined from calibration techniques, and/or from the camera itself, determining the extrinsics is often a challenge ; however, robots in a fixed workcell, or using a camera attached to a manipulator arm with accurate encoders, can readily determine camera extrinsics. This makes NeRF a particularly good match for robot manipulators.

In experiments, we show qualitatively and quantitatively that NeRF-based grasp-planning can achieve high accuracy on NeRF models trained from photo-realistic synthetic images and from real images, and achieve 90 % or better grasp success rates on real objects.

The contributions of this paper are: (1) Integration of NeRF with robot grasp planning, (2) A transparency-aware depth rendering method for NeRF, (3) Experiments on synthetic and real images showing NeRF with Dex-Net generates high-quality grasps, (4) Synthetic and real image datasets with transparent objects for training NeRF models.

Related Work

For robots to interact with transparent objects, they must first be able to detect them. The most recent approaches detecting and recognizing transparent objects are data-driven. Lai et al. , then Khaing et al. , proposed using a Convolutional Neural Network (CNN) to detect transparent objects in RGB images. Recently, Xie et al. developed a transformer-based pipeline used for transparent object segmentation. Other methods rely on deep-learning models to predict the object pose. Phillips et al. trained a random forest to detect the contours of transparent objects for the purpose of pose estimation and shape recovery. Xu et al. proposed a two-stage method for estimating the 6-degrees-of-freedom (DOF) pose of a transparent object with a single RGBD image by replacing the noisy depth values with estimated values and training a DenseFusion-like network structure to predict the object’s 6-DOF pose. Sajjan et al. extend this and incorporate a neural network trained for 3D pose estimation of transparent objects in a robotic picking pipeline, while Zhou et al. train a grasp planner directly on raw images from a light-field camera. Zhu et al. used an implicit function to complete missing depth given noisy RGBD observation of transparent objects. However, these data-driven methods rely on large annotated datasets that are hard to curate, whereas Dex-NeRF does not require any prior dataset.

Neural Radiance Fields

Recently, implicit neural representations have led to significant progress in 3D object shape representation and encoding the geometry and appearance of 3D scenes . Mildenhall et al. presented Neural Radiance Fields (NeRF), a neural network whose input is a 3D coordinate with an associated view direction, and output is the volume density and view-dependent emitted radiance at that coordinate. Due to its view-dependent emitted radiance prediction, NeRF can be used to represent non-Lambertian effects such as specularities and reflections, and therefore capture the geometry of transparent objects. However, NeRF is slow to train and has low data efficiency. Yu et al. proposed Plenoctrees, mapping coordinates to spherical harmonic coefficients, shifting the view-dependency from the input to the output. In addition, Plenoctrees pre-samples the model into a sparse octree structure, achieving a significant speedup in training over NeRF. Deng et al. proposed JaxNeRF, an efficient JAX implementation of NeRF that was able to reduce the training time of a NeRF model from over a day to several hours. Deng et al. add depth supervision to train NeRF 2 to 6 $\times$ faster given fewer training views. In this work, we propose to use NeRF to recover the geometry of transparent objects for the purpose of robotic manipulation.

Robotic Grasping

Traditional robot grasping methods analyze the object shape to identify successful grasp poses . Data-driven approaches learn a prior using labeled data or through self-supervision over many trials in a simulated or physical environment and generalize to grasping novel objects with unknown geometry. Both approaches rely on RGB and depth sensors to generate a sufficiently accurate observation of the target object surface, such as depth maps , point clouds , octrees , or a truncated signed distance function (TSDF) from which it can compute the grasp pose. While various grasp-planning methods use different input geometry to compute grasps, in this paper we propose a method to render a high-quality depth map from a NeRF model to then pass to Dex-Net to compute a grasp. While standard depth cameras have gaps in their depth information that needs to be processed out with hole-filling techniques, the depth map rendering from NeRF is directly usable. It is possible that other grasp-planning techniques may be able to plan grasps from NeRF models.

Problem Statement

We assume an environment that has an array of cameras at fixed known locations, or that the robot can manipulate a camera (e.g., wrist-mounted) to capture multiple images of the scene. Given the environment contains rigid transparent objects, compute a frame for a robot gripper that will result in a stable grasp of a transparent object.

Method

In this section, we provide a brief background on NeRF, then describe recovering geometry of transparent objects, integrating with grasp analysis, and improving performance with additional lights.

NeRF learns a neural scene representation that maps a 5D coordinate containing a spatial location $(x,y,z)$ and viewing direction $(\theta,\phi)$ to the volume density $\sigma$ and RGB color $\mathbf{c}$ . Training NeRF’s multilayer perceptron (MLP) requires multi-view RGB images of a static scene with their corresponding camera poses and intrinsic parameters. The expected color $C(\mathbf{r})$ of the camera ray $\mathbf{r}=\mathbf{o}+t\mathbf{d}$ between near and far scene bounds $t_{n}$ and $t_{f}$ is:

where $T(t)=\exp\left(-\int_{t_{n}}^{t}\sigma(\mathbf{r}(s))ds\right)$ is the probability that the camera ray travels from near bound $t_{n}$ to point $t$ without hitting any surface. NeRF approximates the expected color $\hat{C}(\mathbf{r})$ as:

where $T_{i}=\exp\left(-\sum^{i-1}_{j=1}\sigma_{j}\delta_{j}\right)$ and $\delta_{i}=t_{i+1}-t_{i}$ is the distance between consecutive samples on the ray $\mathbf{r}$ . During training, NeRF minimizes the error between the rendered and ground truth rays’ colors using gradient descent.

2 Recovering Geometry of Transparent Objects

We observe that NeRF does not directly support transparent object effects—it casts a single ray per source image pixel without reflection, splitting, or bouncing. The incorporation of the viewing-direction in its regression and supervising with view-dependent emitted radiance allows recovery of non-Lambertian effects such as reflections from a specular surface. However, while RGB color $\mathbf{c}$ is view-dependent, the volume density $\sigma$ is not—meaning NeRF has to learn a non-zero $\sigma$ to represent any color at that spatial location. Visually, the usual result is that the transparent object shows up as a “ghostly” or “blurry” version of the original object.

When training, a NeRF model learns a density $\sigma$ of each spatial location. This density corresponds to the transparency of the point, and serves to help learn how much a spatial location contributes to the color of a ray cast through it. Although NeRF converts each $\sigma_{i}$ to an occupancy probability $\alpha_{i}=1-\exp(-\sigma_{i}\delta_{i})$ , where $\delta_{i}$ is the distance between integration times along the ray, thus implicitly giving $\alpha_{i}$ an upper bound of 1, it does not place a bound on the raw $\sigma$ value. We use the raw value of $\sigma$ to determine if a point in space is occupied.

3 Rendering Depth for Grasp Analysis

We propose using Dex-Net to compute grasps on transparent objects. Dex-Net computes candidate grasp poses given a depth image of a scene. To generate a depth image, we consider two candidate reconstructions of depth. First, we use the same rendering of sampled points along a camera ray that NeRF uses. This Vanilla NeRF reconstruction first converts $\sigma_{i}$ to an occupancy probability $\alpha_{i}$ . It then applies the transformation $w_{i}=\alpha_{i}\prod_{j=1}^{i-1}\left(1-\alpha_{j}\right)$ . To render depth at pixel coordinate $[u,v]$ , it computes the sum of sample distances from the camera weighted by the termination probability $D[u,v]=\sum_{i=1}^{N}w_{i}\delta_{i}$ . When applied on transparent objects, however, this results in noisy depth maps, as shown in Fig. 3.

Instead, we consider a second, transparency-aware method that searches for the first sample along the ray for which $\sigma_{i}>m$ where $m$ is a fixed threshold. The depth is then set to the distance of that sample $\delta_{i}$ . We explore different values for $m$ , and observe that low values result in a noisy depth map while high values create holes in the depth map. In our experiments we set $m=15$ (see Fig. 8).

4 Improving Reconstruction with Light Placement

For NeRF to learn the geometry of a transparent object, it must be able to “see” it from multiple camera views. If the transparent object is not visible from any views, then it will have no effect on the loss function used in training, and thus not be learned. We thus look for a way to improve visibility of transparent objects to NeRF.

One property that transparent objects share (e.g., glass, clear plastic) is that they are glossy and thus produce specular reflections when the camera view direction is opposite to the surface normal of the incident direction of light. To the NeRF model, a specular reflection viewed from multiple angles will appear as a white point on a solid surface—i.e., $\mathbf{c}=^{T}$ and $\sigma>0$ , while from other angles it will appear as $\sigma\leq 0$ . As $\sigma$ is view-independent, NeRF learns a $\sigma$ between fully opaque and fully transparent for such points.

By placing additional lights in the scene, we create more angles from which cameras will see specular reflections from transparent objects—this results in NeRF learning a model that fills holes in the scene. While the number and placement of lights for optimal training is dependent on both the expected object distribution and camera placement, in experiments (Sec. 5.5) we show that increasing from 1 light to a 5x5 array of lights improves the quality of the learned geometry.

Experiments

We experiment in both simulation and on a physical ABB YuMi robot. We generate multiple datasets, where each dataset consists of images and associated camera transforms of one static scene containing one or more transparent objects. We train NeRF models using a modified JaxNeRF implementation on 4 Nvidia V100 GPUs. We use an existing pre-trained Dex-Net model for grasp planning without modification or fine-tuning. We can do this since NeRF models can be rendered to depth maps from arbitrary camera intrinsics and extrinsics, thus we match our NeRF rendering to the Dex-Net model instead of training a new one.

As existing NeRF datasets do not include transparent objects, and existing transparent-object-grasping datasets do not include multiple camera angles, we generate new datasets using 3 different methods: synthetic, Cannon EOS 60D camera with a Tamron Di II lens with a locked focal length, and an Intel RealSense.

For synthetic datasets, we use Blender 2.92’s physically-based Cycles renderer with path tracing set to 10240 samples per pixel, and max light path bounces set to 1024. We chose theses settings by increasing them until renderings were indistinguishable from the previous setting—finding that lower settings lead to dark regions and smaller specular reflections. For glass materials, we set the index of refraction to 1.45 to match physical glass. We include 8 synthetic datasets of transparent objects: 2 scenes with clutter: light array and single light; 4 singulated objects from Dex-Net: Pipe Connector, Pawn, Turbine Housing, Mount; and 2 household objects: Wineglass upright and Wineglass on side. As these computationally demanding to render due to the high quality settings, we distribute these as part of the contribution.

For the Cannon EOS and RealSense real-world datasets, we place ArUco markers in the scene to aid in camera pose recovery and take photos around the objects using a fixed ISO, f-stop, and focal length. We use bundle adjustment from COLMAP to refine the camera poses and intrinsics to high accuracy. We include 8 physical datasets of transparent objects with a variety of camera poses: table with clutter, Dishwasher, Tape Dispenser, Wineglass on side, Flask, Safety Glasses, Bottle upright, Lion Figurine in clutter. The main difficulty in generating these datasets is calibration and computing high-precision camera poses.

The datasets, available at https://sites.google.com/view/dex-nerf differ from other datasets in their focus on scenes with transparent objects in a graspable setting, with over 70 camera poses each.

2 Synthetic Grasping Experiments

$10<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mn>20</mn></mrow><annotation encoding="application/x-tex">20</annotation></semantics></math>2030<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mn>40</mn></mrow><annotation encoding="application/x-tex">40</annotation></semantics></math>4050<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mn>60</mn></mrow><annotation encoding="application/x-tex">60</annotation></semantics></math>6070<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mn>80</mn></mrow><annotation encoding="application/x-tex">80</annotation></semantics></math>8090<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mn>100</mn></mrow><annotation encoding="application/x-tex">100</annotation></semantics></math>10020<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mn>40</mn></mrow><annotation encoding="application/x-tex">40</annotation></semantics></math>4060<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mn>80</mn></mrow><annotation encoding="application/x-tex">80</annotation></semantics></math>80100$ Training epochs ( $\times 1000$ )Grasp Success Rate (%)Pipe ConnectorPawnTurbine HousingMount We also test the ability of Dex-NeRF to generate grasps on the synthetic singulated transparent Dex-Net object datasets. For each dataset, we evaluate the grasp in simulation using a wrench resistance metric measuring the ability of the grasp to resist gravity . Fig. 4 shows images of the synthetic objects, Dex-NeRF-generated depth map, and an example sampled grasp for each. To measure the effect of training time on grasp-success rate, we simulate and record grasps over the course of training. In Fig. 5, we observe that grasp success rate improves with training time, but plateaus between 80 % and 98 % success rate at around 50k to 60k iterations. This suggests that there may be a practical fixed iteration limit to obtain high grasp success rates.

We test Dex-NeRF on a scene of a tabletop cluttered with transparent objects. In this experiment, the goal is to grasp a transparent object placed in a stable pose in close proximity to other transparent objects. The challenge is twofold: the depth rendering quality should be sufficient for both grasp planning and collision avoidance. Fig. 1 shows the robot and scene in the upper left, and the overhead image, depth, and computed grasp inline in the pipeline, and the final computed grasp with simulated execution is in the upper middle image. The final grasp contact point was accurate to a 2 mm tolerance, suggesting that Dex-NeRF with sufficient images taken from precisely-known camera locations may be practical in highly clutter environments.

3 Physical Grasping Experiments

To test the Dex-NeRF in a physical setup, we place transparent singulated objects in front of an ABB YuMi robot, and have the robot perform the computed grasps. We compare to 2 baselines: (1) PhoXi, in which a PhoXi camera provides the depth map; and (2) Vanilla NeRF, in which we use the original depth rendering from NeRF. The PhoXi camera is normally able to generate high-precision depth maps for non-transparent objects. All methods use the same pre-trained Dex-Net model, and both Vanilla NeRF and Dex-NeRF use the same NeRF model—the only difference is the depth rendering. We test with 6 objects (Fig. 6), and compute and execute 10 different grasps for each and record the success rate. A grasp is successful if the robot lifts the object. In Table 1, we see that Dex-NeRF gets 90 % and 100 % success rates for all objects, while the baselines get few successful grasps. The PhoXi camera is unable to recover any meaningful geometry which causes Dex-Net predictions to fail. The Vanilla NeRF depth maps often have unpredictable protrusions that result in Dex-Net generating unreliable grasps.

4 Comparison to RealSense Depth

We qualitatively compare the the rendered depth map of the proposed pipeline against a readily-available depth sensor on scenes with transparent objects in real-world settings. We select the Intel RealSense as it is common to robotics applications, readily available, and high-performance. The RealSense, like most stereo depth cameras, struggles with transparent objects as they are unable to compute a stereo disparity between pixels from different cameras when the pixels are specular reflections or the color of the object behind the transparent object. The RealSense optionally projects a structured light pattern on the the scene to aid in computing depth from textureless surfaces, however, in experiments we observed no qualitative difference with and without the light pattern emitter enabled. To run this experiment, we use a Canon EOS as described in the main text, and use a RealSense to take a depth picture of the scene. In this experiment, we observe that the RealSense is unable to compute the depth of most transparent objects, and often produces regions of unknown depth (shown in black) where transparent objects are. On the other hand, the proposed pipeline produces high-quality depth maps with only a few areas of noise. The results of these experiments is shown in Fig. 2.

5 One vs Many Lights

We experiment with different light setups to test the effect of specular reflections on the ability of NeRF to recover geometry of transparent objects. We create two scenes (Fig. 7), one with a single bright light source directly above the work surface, and another with an array of 5x5 (25) lights above the work surface. We set the total wattage of the lights in each scene to be the same. Since most lights in the multiple light scene are further away from the work surface than the single light source, the scene appears darker, though more evenly illuminated. The effect of the specular reflections is prominent on the lightbulb in the lower part of the image. In the single light source, there is a single specular reflection, while in the multiple light scene, the reflection of the array of lights is visible.

With the same camera setup for both scenes, we train NeRF models with the same number of iterations. We show the depth rendering in Fig. 7 and circle a glass and a wineglass on their side. In the single-light source image, the closer surfaces of the glasses are missing, while in the multiple-light source depth image, the glasses are nearly fully recovered. This suggests that additional lights in the scene can help NeRF recover the geometry of transparent objects better.

6 Workcell Setup

We experiment with a potential setup for a robot workcell in which a grid of overhead cameras captures views of the cluttered scene so that a robot manipulator arm can then perform tasks with transparent objects in the workcell. We propose that a grid of overhead cameras would be practical to setup and would not obstruct manipulator tasks nor operator interventions. The objective is to determine how many overhead cameras would be needed to recover a depth map of sufficient accuracy to perform maniplation tasks.

We place a 2 m by 2 m grid of cameras 1 m above the work surface, and have them all point at the center of the work surface. Each camera has the same intrinsics, and are evenly spaced along the grid. We experiment with grids having 4, 9, 16, 25, 36, and 49 cameras. The environment has the same 5x5 grid of lights as before. For each camera grid, we train JaxNeRF for 50k iterations and compare performance.

After training, we observe increasing peak signal to noise ratios (PSNR) and structural similarity (SSIM) scores with increasing number of cameras. The 2x2 grid of cameras produces a high train-to-test ratio for PSNR, likely indicating overfitting to training data, and results in a depth map without apparent geometry. This ratio decreases with additional cameras. The minimum number of cameras for this proposed setup appears to be around 9 (3x3) as its depth map is usable for grasp planning, while the 5x5 grid shows better PSNR and SSIM and ratio between train and test PSNR, and the 7x7 grid is the best. See Fig. 9 for a visual comparison. Additionally, we trained 9x9, 11x11, and 13x13 grids, observing no statistically significant improvement beyond the 7x7 grid.

Conclusion

In this work, we showed that NeRF can recover the geometry of transparent objects with sufficient accuracy to compute grasps for robot manipulation tasks. NeRF learns the density of all points in space, which corresponds to how much the view-dependent color of each point contributes to rays passing through it. With the key observation that specular reflections on transparent objects cause NeRF to learn a non-zero density, we recover the geometry of transparent objects through a combination of additional lights to create specular reflections and thresholding to find transparent points that are visible from some view directions. With the geometry recovered, we pass it to a grasp planner, and show that the recovered geometry is sufficient to compute a grasp, and accurate enough to achieve 90 % and 100 % grasp success rates in physical experiments on an ABB YuMi robot. We created synthetic and real datasets for experiments in transparent geometry recovery, but we believe these datasets may be of interest to researchers interested in extending NeRF capabilities in other ways and thus contribute them as part of this research project. Finally, to test if NeRF could be used in a robot workcell, we experimented with grids of cameras facing a worksurface and their ability to recover geometry in potential setup, and showed the increased capabilities and point of diminishing return for additional cameras.

In future work, we hope to address one of the main drawbacks of NeRF—the long training time required to obtain a NeRF model. Many research groups have sped up training time through improved implementations, new algorithms, new network architectures, pre-conditioned network weights, focused sampling, and more. While these approaches apply to general NeRF training, we plan to exploit features specific to robot scenerios to speed up training, including using depth camera data as additional training data, manipulator-arm-mounted cameras to inspect regions of interest, and visio-spatial foresight to adapt to changes in the environment.

Acknowledgments

This research was performed at the AUTOLAB at UC Berkeley in affiliation with the Berkeley AI Research (BAIR) Lab, Berkeley Deep Drive (BDD), the Real-Time Intelligent Secure Execution (RISE) Lab, and the CITRIS “People and Robots” (CPAR) Initiative. We thank our colleagues who provided helpful feedback and suggestions, in particular Matthew Tancik. This article solely reflects the opinions and conclusions of its authors and do not reflect the views of the sponsors or their associated entities.