3D Object Detection and Pose Estimation of Unseen Objects in Color Images with Local Surface Embeddings

Giorgia Pitteri, Aurélie Bugeau, Slobodan Ilic, Vincent Lepetit

Introduction

Deep Learning (DL) provides powerful techniques to estimate the 6D pose of an object from color images, and impressive results have been achieved over the last years , including in the presence of occlusions , in the absence of texture, and for objects with symmetries (which create pose ambiguities) . However, most of recent works focus on supervised approaches, which require that for each new object, these methods have to be retrained on many different registered images of this object. Even if domain transfer methods allow for training such methods on synthetic images instead of real ones at least to some extent, such training sessions take time, and avoiding them is highly appealing.

Recently, a few methods were proposed for 3D pose estimation for objects that were not seen during training, only exploiting an untextured CAD model provided for the new objects. This is an important problem in industrial contexts, but also very challenging as aligning an untextured CAD model to a color image remains very difficult, especially without pose prior. In DeepIM , the authors propose a pose refiner able to perform such alignment given some initial pose, however it has been demonstrated on very simple synthetic images with constant lighting. proposes to learn to detect corners by using training images of a small set of objects, and estimates the object pose by robustly matching the corners of the CAD model with the corners detected in the input image. However, this requires the object to have specific corners and a skilled user to select the corners on the CAD model. Very recently, proposes a single-encoder-multi-decoder network to predict the 6D pose of even unseen objects. Their encoder can learn an interleaved encoding where general features can be shared across multiple instances. This leads to encodings that can represent object orientations from novel, untrained instances, even when they belong to untrained categories. Even if the idea is promising, to achieve competitive results, they need to use depth information and refine the pose with an ICP algorithm.

In this paper, we investigate 6D object pose estimation in an industrial scenario with the challenges this implies: We want to handle symmetrical, textureless, ambiguous, and unseen objects, given only their CAD models. By contrast with some previous works, we also do not assume that the ground truth 2D bounding boxes for the objects are available. As shown in Figure 1, our approach combines machine learning and 3D geometry: Like previous works , we establish dense correspondences between the image locations and 3D points on the CAD model, as they showed that this yields to accurate poses. However, there is a fundamental difference between these works and ours: They can train a machine learning model in advance to predict the 3D coordinates of the pixels in a given image. In our case, we want to avoid any training phase for new objects. We therefore rely on a different strategy: We introduce an embedding capturing the local geometry of the 3D points lying on the object surface. Given a training set for a small number of objects, we learn to predict these embeddings per pixel for images of new objects. By matching these embeddings with the embeddings computed for 3D points on the object surface, we get 2D-3D correspondences from which we estimate the object’s 6D pose using RANSAC and a P $n$ P solver.

This approach is conceptually simple, robust to occlusions, and provides an accurate 6D pose. However, to be successful, some special care is needed. First, the embeddings need to be rotation invariant. Second, because of the symmetries and this rotation invariance, many correspondences between pixels and 3D points are possible a priori and the complexity of finding a set of correct correspondences can become exponential. We control this complexity in two ways. We focus on image locations with the most discriminative embeddings as they have less potential correspondences. We also observe that Mask R-CNN is able to predict the masks of new objects when trained without any class information, and thus can segment new objects without re-training. We use this to constrain the sets of correspondences in RANSAC to lie on the same mask, and thus drastically decrease the number of samples to consider in RANSAC.

In the remainder of the paper, we review the state-of-the-art on 3D object pose estimation from images, describe our method, and evaluate it on the T-LESS dataset , which is made of very challenging objects and sequences.

Related Work

In this section, we first review recent works on 3D object detection and pose estimation from color images. We also review methods for using synthetic images for training as it is a popular solution for 3D pose estimation. Finally, we review the few works that consider the same problem as us.

The use of Deep Learning has recently significantly improved the performance of 6D pose estimation algorithms. Different general approaches have been proposed. One approach is to first estimate 2D bounding boxes for the visible objects, and predict the 6D pose of each object directly from the image region in the bounding box . The pose can be predicted directly using quaternions for the rotation example, or via 3D points or the reprojections of 3D points related to the object, or by learning a code book using AutoEncoders. This last method has the advantage to work well with symmetrical objects, which are common in industrial contexts. Another approach, aiming to be more robust to occlusions, is to predict for each pixel offsets to the reprojections of 3D points related to the object .

Closer to our own approach, several works first predict for each pixel its 3D coordinates in the object’s coordinate frame . This yields 2D-3D correspondences from which the object’s 3D pose can be estimated using a P $n$ P algorithm, possibly together with RANSAC for more robustness. In our case, we cannot directly predict the 3D coordinates of pixels as it can be done only for the objects or categories used for training. Instead, we learn to predict an embedding for the 3D local geometry corresponding to each pixel, and we rely on this embedding to match the pixel to its corresponding 3D point on the CAD models of new objects.

2 Training on synthetic images for 6D pose estimation

One popular approach to 6D pose estimation given only a CAD model and no, or few, real training images is to exploit synthetic images. There is however a domain gap between real and synthetic images, which has to be considered to make sure the method generalizes well to real images.

A very simple approach is to train a convolutional network for some problem such as 2D detection on real images and use the first part of the network for extracting image features . Then, a network taking these features as input can be trained on synthetic images. This is easy to do, but it is not clear how many layers should be used exactly. Generative Adversarial Networks (GANs) and Domain Transfer have been used to make synthetic images more realistic . Another interesting approach is domain randomization , which generates synthetic training images with random appearance by applying drastic variations to the object textures and the rendering parameters to improve generalization.

These works can exploit CAD models for learning to detect new objects, however they also require a training phase for new objects. In this work, we do not need such phase.

3 6D pose estimation without retraining

Very few recent works already tackled 6D pose estimation without retraining for new objects. One early approach targeting texture-less objects is to rely on templates . Deep Learning has also been applied to such problem, by learning to compute a descriptor from pairs or triplets of object images . Like ours, these approaches do not require re-training, as it only requires to compute the descriptors for images of the new objects. However, it requires many images from points of view sampled around the object. It may be possible to use synthetic images, but then, some domain transfer has to be performed. But the main drawback of this approach is the lack of robustness to partial occlusions, as the descriptor is computed for whole images of objects. It is also not clear how it would handle ambiguities, as it is based on metric learning on images. In fact, such approach has been demonstrated on the LineMod, which is made of relatively simple objects, and never on the T-LESS dataset, which is much more challenging.

More recently, DeepIM proposed a pose refiner able to refine a given initial pose. In , this refiner was applied to new objects, but only on very simple synthetic images with constant lighting. proposes to learn to detect corners by using training images of a small set of objects and estimates the object pose by robustly matching the corners of the CAD model with the corners detected in the input image. This method requires objects to have specific corners and to offline select corners on the CAD model. Even more recently, proposes an extension of able to generalize to new objects. Thanks to the single-encoder-multi-decoder architecture, they are able to learn an interleaved encoding where general features can be shared across multiple instances of novel categories. To achieve competitive results, they need to use depth information and refine the pose with an ICP algorithm.

Our approach is related to , but considers any 3D location on the objects to get matches, not only corners. We compare against and in the experimental section.

Method

We describe our approach in this section. We first explain how we compute the local surface embeddings and how we obtain correspondences between the CAD models and the images. We then describe our pose estimation algorithm.

To match new images with CAD models, we rely on embeddings of the local surfaces of the objects. To be able to match these embeddings under unknown poses, they need to be translation invariant and rotation invariant. Achieving translation invariance is straightforward, since we consider the local geometry centered on 3D points. Achieving rotation invariance is more subtle, especially because of ambiguities arising in practice with symmetrical objects. This is illustrated in Figure 2(b): We need to compute the same embeddings for local geometries that are similar up to a 3D rotation.

More exactly, given a 3D point $\mathbf{P}$ on the surface of an object, we define the local geometry as the set of 3D points $\mathbf{M}_{n}$ in a spherical neighborhood centered on $\mathbf{P}$ and of radius $r$ . In practice, on T-LESS, we use $r=3cm$ . To compute a rotation-invariant embedding, we transform these points from the object coordinate system to a local patch coordinate system using a rotation matrix computed from the decomposition of the covariance matrix of the 3D points $\mathbf{M}_{n}$ after centering on $\mathbf{P}$ :

where $\mathbf{v}_{n}=(\mathbf{M}_{n}-\mathbf{P})$ using a Singular Value Decomposition (SVD):

$R$ is an orthogonal matrix, but not necessarily a rotation matrix, and small differences in the local geometry can result in very different values for $R$ . We therefore apply a transformation to $R$ to obtain a new matrix $\bar{R}$ so that $\bar{R}$ is a suitable rotation matrix. It can be checked that applying $\bar{R}$ to the $\mathbf{v}_{i}$ vectors will achieve rotation invariance for the local surface embeddings.

Let’s denote by $r_{1}$ , $r_{2}$ , and $r_{3}$ the rows of $R$ , and by $\bar{r}_{1}$ , $\bar{r}_{2}$ , and $\bar{r}_{3}$ the rows of $\bar{R}$ . Applying $R$ to the normal $n$ of the object’s surface at $\mathbf{P}$ yields a 3-vector $R\cdot n$ close to either $^{\top}$ or $^{\top}$ , depending on the orientation of $R$ selected for the SVD. For normalisation, we choose that $\bar{R}\cdot n$ should always be closer to $^{\top}$ . We therefore compute $o=r_{3}^{\top}\cdot n$ . If $o$ is positive, we take $\bar{r}_{3}=r_{3}$ , otherwise we take $\bar{r}_{3}=-r_{3}$ . As a result, $\bar{R}\cdot n$ is always closer to $^{\top}$ that to $^{\top}$ . Finally, we take $\bar{r}_{1}=r_{1}$ and $\bar{r}_{2}=-\bar{r}_{1}\wedge\bar{r}_{3}$ , where $\wedge$ denotes the cross-product, which ensures that $\bar{R}$ is a rotation matrix.

We explain now how we define the local surface embeddings. For our experiments, we use the local moments of the local 3D points for simplicity but any other embeddings such as could also work. Let us denote by $[x_{n},y_{n},z_{n}]$ the vectors $\bar{R}\mathbf{v}_{n}$ , then local surface embeddings can be computed as:

where $w_{n}=\exp(-\|\mathbf{v}_{n}\|^{2}/\sigma^{2})$ is a weight associated to each point based on its distance from $\mathbf{P}$ (we use $\sigma=5$ in practice) and $i,j,k$ are exponents in the range $ $. Theoretically it is possible to take all the combinations of exponents but we empirically found that the most discriminative values are computed using:$ i\in\{0,2\} $,$ j\in\{0,2\} $,$ k\in\{0,1,2\} $, which gives 11 values for the full vector$ \text{LSE}(\mathbf{P}) $as taking$ i=j=k=0 $gives a constant value and is not useful. Finally, we normalize the values of$ \text{LSE}(\mathbf{P})$ to zero mean and unit variance so they have similar ranges. Figure 3 displays the embeddings for an example image.

2 Predicting the local surface embeddings for new images

Given a new CAD model, it is trivial to compute the local surface embeddings on points on its surface. Given a new input image, we would like to also compute the embeddings for the object points visible in this image. We use a deep network to perform this task. To do so, we create a training set by generating many synthetic images of known objects under various poses. We also compute the LSEs for all the pixels corresponding to a 3D point of one of the objects. We then train a U-Net-like architecture to predict the LSEs given a color image. More details on the architecture and its training are provided in the experimental section.

This training is done once, on known objects, but because the embeddings depend only on the local geometry, the network generalizes well to new objects, as shown in Figure 4.

3 Pose Estimation Algorithm

The pseudocode for our detection and pose estimation algorithm is given as Algorithm 1. Given a new image, we compute the LSEs for each of its pixels using the network described in Section 3.2 and establish correspondences between image pixels and object 3D points. However, the number of possible correspondences can quickly become very large, which would yield a combinatorial explosion in the number of set of correspondences needed in RANSAC. We control the complexity in two different ways.

First, we focus on the most discriminative embeddings. Points on planar regions are very common and would generate many correspondences. We discard them by thresholding the embedding values: Points with very low absolute embedding values for the LSEs are removed. Figure 5 (a) shows how pixels are selected.

Second, we force the correspondences in each sample considered by RANSAC to belong to the same object. Even when objects are not known in advance, it is possible to segment them: To do so, we use Mask-RCNN to predict the masks of the objects. We fine-tuned it on our synthetic images already used for training the LSE predictor, as described in Section 3.2 in a class-agnostic way since we want to generalize to new objects. We found out that it works very well with new objects even for cluttered backgrounds, as shown in Figure 6. This also allows us to easily discard pixels on the background from the possible correspondences.

We match the embeddings predicted for the pixels of the input image against the embeddings computed for the 3D points on the CAD model based on their Euclidean distances. In our implementation, we use the FLANN library to efficiently get the $k$ nearest neighbors of a query embedding. In practice, we use $k=100$ . This usually returns points in several clusters, as close points tend to have similar embeddings. We therefore go through the list of nearest neighbors sorted by increasing distances. We keep the first 3D point and remove from the list the other points that are also close to this point, and we iterate. This provides for each pixel a list of potential corresponding 3D points separated from each other.

When working on industrial objects like the ones in T-LESS, some pixels can be matched with several 3D points, as shown in Figure 5(b), because of the rotation invariance property of the local LSEs and the similarities between local parts of different objects.

We finally use LO-RANSAC (Locally Optimized RANSAC) with a P $n$ P algorithm (we use followed by a Levenberg-Marquardt optimization) to compute the poses of the visible objects. We take random $n\in[6;10]$ for each RANSAC sample. At each iteration, we compute a score for the predicted pose as a weighted sum of the Intersection-over-Union between the mask from Mask-RCNN and the mask obtained by rendering the model under the estimated pose, and the Euclidean distances between the predicted LSEs and the LSEs for the CAD model after reprojection. We keep the pose with the largest score and refine it using all the inlier correspondences to obtain the final 6D pose.

Evaluation

In this section, we present and discuss the results of our pose estimation algorithm on the challenging T-LESS dataset , made of texture-less, ambiguous and symmetrical objects with no disciminative parts. It is well representative of the problems encountered in industrial context.

To train our LSE prediction network, we generate synthetic images using the CAD models provided with T-LESS for a subset of objects in this dataset. The exact subset depends on the experiment, and we will detail them below. Similar to the BlenderProc4BOP introduced in the BOP challenge , these images are created with Cycles, a photorealistic rendering engine of the open source software Blender by randomly placing the training objects in a simple scene made of a plane randomly textured and randomly lighted. We used both these synthetics and real images for training the network combined with data augmentation to take care of the domain gap between our synthetic images and the real test images. More specifically, we use $15K$ synthetic images and $\sim 7K$ real images—all the training images provided by T-LESS for the objects that are used for training the LSE prediction and Mask-RCNN. To create the ground truth embeddings, for each training image, we backproject the pixels lying on the objects to obtain their corresponding 3D points and their LSEs. Neither the embedding prediction network nor Mask-RCNN see the test objects during training.

2 LSE prediction network architecture and training

The architecture of the network predicting the LSEs for a given input image is a standard U-Net-like encoder-decoder convolutional neural network taking a $720\times 540$ RGB image as input. The encoder part is a 12-layer ResNet-like architecture; the decoder upsample the feature maps up to the original size using bilinear interpolations followed by convolutional layers. We train the network with the Adam optimizer and a learning rate set to $10^{-4}$ . We also use batch normalization to ensure good convergence of the model. Finally, the batch size is set to 8 and we train the network for 150 epochs.

3 Metrics

We evaluate our method using several metrics from the literature. Analogously to other related papers , we consider the percentage of correctly predicted poses for each sequence and each object of interest, where a pose is considered correct based on the ADD metric (or the ADI metric for symmetrical objects) . This metric is based on the average distance in 3D between the model points after applying the ground truth pose and the estimated one. A pose is considered correct if the distance is less than 10% of the object’s diameter.

Following the BOP benchmark , we also report the Visible Surface Discrepancy (VSD) metric. The VSD metric evaluates the pose error in a way that is invariant to the pose ambiguities due to object symmetries. It is computed from the distance between the estimated and ground truth visible object surfaces in the following way:

where $\hat{S}$ and $\bar{S}$ are distance maps obtained by rendering the object model in the estimated and ground-truth poses respectively. The distance maps are compared with the distance map $S_{I}$ of the test image $I$ to obtain the visibility masks $\hat{V}$ and $\bar{V}$ , i.e. the sets of pixels where the object model is visible in image $I$ . We report the mean VSD recall of 6D object poses at $err_{\text{VSD}}<0.3$ with tolerance $\tau=20mm$ and $>10\%$ object visibility.

4 Results

The complexity of the test scenes in T-LESS varies from several isolated objects on a clean background to very challenging ones with multiple instances of several objects with a high amount of occlusions and clutters. We compare our method against the two works that already consider 6D object detection and pose estimation for unknown objects on T-LESS, CorNet and the MP-Encoder method of . As the codes for these two methods are not available at the time of writing, we use the same protocols as in these works and report the results from the papers.

We use here the same protocol as in : We split the objects from T-LESS into two sets: One set of known objects (#6, #19, #25, #27, and #28) and one set of unknown objects (#7, #8, #20, #26, and #29), and we compare the 3D detection and pose estimation performance of our method and CorNet for the unknown objects in T-LESS test scenes #02, #03, #04, #06, #08, #10, #11, #12, #13, #14, and #15. We use synthetic images of the known objects for training the LSE prediction network. Results are reported in Table 1. We outperform CorNet on most of the objects, except on objects #7 and #8. This is because these objects have some 3D points with local geometry very different from the training objects (at the connections of the different parts). As a result, the predicted LSEs for these parts are not very accurate, generating wrong matches. Figured 8 shows some qualitative results for the unknown objects in the test images.

Comparison with MP-Encoder [14].

We use here the same protocol as in : The objects from T-LESS are split into a set of known objects (#1-#18) and one set of unknown objects (#19-#30), and we compare the 3D detection and pose estimation performance of our method and MP-Encoder for the unknown objects in T-LESS test scenes following the BOP benchmark . We use synthetic images of the known objects for training the LSE prediction network. Note that we report here the number of Table 3 from the paper as the other reported results assume that the ground truth bounding boxes, the ground truth masks, or depth information are provided. Results are reported in Table 2. While our method performs slightly better, the performances are close and tells us that both methods are promising. The main difference is that the MP-Encoder relies on an embedding completely learnt by a network while our method incorporates some geometrical meaning that makes our approach more appealing for industrial purposes. Note that, as shown in Fig. 7, our LSE network can handle objects with rounded shape without being limited to objects with prominent corners as .

5 Robustness to Texture

Our focus is on untextured objects, as industrial objects typically do not exhibit textures like the T-LESS objects. However, our LSEs can be predicted for textured objects as well. To show this, we retrained our LSE prediction network on synthetic images of the T-LESS objects rendered with random textures. The LSEs prediction for some test images are shown in the bottom row of Fig. 7.

Conclusion

We introduced a novel approach for the detection and the 3D pose estimation of industrial objects in color images. It only requires the CAD models of the objects and no retraining is needed for new objects. We introduce a new type of embedding capturing the local geometry of the 3D points lying on the object surface and we train a network to predict these embeddings per pixel for images of new objects. From these local surface embeddings, we establish correspondences and obtain the pose with a P $n$ P+RANSAC algorithm. Describing the local geometries of the objects allows to generalize to new categories and the rotation invariance of our embeddings makes the method able to solve typical ambiguities that raise with industrial and symmetrical objects. We believe that using local and rotation invariance descriptors is the key to solve the 6D pose of new textureless objects from color images.