Deep Global Registration

Introduction

A variety of applications, including 3D reconstruction, tracking, pose estimation, and object detection, invoke 3D registration as part of their operation . To maximize the accuracy and speed of 3D registration, researchers have developed geometric feature descriptors , pose optimization algorithms , and end-to-end feature learning and registration pipelines .

In particular, recent end-to-end registration networks have proven to be effective in relation to classical pipelines. However, these end-to-end approaches have some drawbacks that limit their accuracy and applicability. For example, PointNetLK uses globally pooled features to encode the entire geometry of a point cloud, which decreases spatial acuity and registration accuracy. Deep closest point makes strong assumptions on the distribution of points and correspondences, which do not hold for partially overlapping 3D scans.

In this work, we propose three modules for robust and accurate registration that resolve these drawbacks: a 6-dimensional convolutional network for correspondence confidence estimation, a differentiable Weighted Procrustes method for scalable registration, and a robust $\text{SE}(3)$ optimizer for fine-tuning the final alignment.

The second component we develop is a differentiable Weighted Procrustes solver. The Procrustes method provides a closed-form solution for rigid registration in $\text{SE}(3)$. A differentiable version of the Procrustes method by Wang et al. has been used for end-to-end registration. However, the differentiable Procrustes method passes gradients through coordinates, which requires $O(N^{2})$ time and memory for $N$ keypoints, limiting the number of keypoints that can be processed by the network. We use the inlier probabilities predicted by our first module (the 6D convolutional network) to guide the Procrustes method, thus forming a differentiable Weighted Procrustes method. This method passes gradients through the weights associated with correspondences rather than correspondence coordinates. The computational complexity of the Weighted Procrustes method is linear in the number of correspondences, allowing the registration pipeline to use dense correspondence sets rather than sparse keypoints. This substantially increases registration accuracy.

Our third component is a robust optimization module that fine-tunes the alignment produced by the Weighted Procrustes solver. This optimization module minimizes a differentiable loss via gradient descent on the continuous $\text{SE}(3)$ representation space . The optimization is fast since it does not require neighbor search in the inner loop .

Experimentally, we validate the presented modules on a real-world pairwise registration benchmark and large-scale scene reconstruction datasets . We show that our modules are robust, accurate, and fast in comparison to both classical global registration algorithms and recent end-to-end approaches . All training and experiment scripts are available at https://github.com/chrischoy/DeepGlobalRegistration.

Related Work

We divide the related work into three categories following the stages of standard registration pipelines that deal with real-world 3D scans: feature-based correspondence matching, outlier filtering, and pose optimization.

Feature-based correspondence matching. The first step in many 3D registration pipelines is feature extraction. Local and global geometric structure in 3D is analyzed to produce high-dimensional feature descriptors, which can then be used to establish correspondences.

Traditional hand-crafted features commonly summarize pairwise or higher-order relationships in histograms . Recent work has shifted to learning features via deep networks . A number of recent methods are based on global pooling models , while others use convolutional networks .

Our work is agnostic to the feature extraction mechanism. Our modules primarily address subsequent stages of the registration pipeline and are compatible with a wide variety of feature descriptors.

Outlier filtering. Correspondences produced by matching features are commonly heavily contaminated by outliers. These outliers need to be filtered out for robust alignment. A widely used family of techniques for robust model fitting is based on RANdom SAmple Consensus (RANSAC) , which iteratively samples small sets of correspondences in the hope of sampling a subset that is free from outliers. Other algorithms are based on branch-and-bound , semi-definite programming , and maximal clique selection . These methods are accurate, but commonly require longer iterative sampling or more expensive computation as the signal-to-noise ratio decreases. One exception is TEASER , which remains effective even with high outlier rates. Other methods use robust loss functions to reject outliers during optimization .

Our work uses a convolutional network to identify inliers and outliers. The network needs only one feed-forward pass at test time and does not require iterative optimization.

Pose optimization. Pose optimization is the final stage that minimizes an alignment objective on filtered correspondences. Iterative Closest Points (ICP) and Fast Global Registration (FGR) use second-order optimization to optimize poses. Makadia et al. propose an iterative procedure to minimize correlation scores. Maken et al. propose to accelerate this process by stochastic gradient descent.

Recent end-to-end frameworks combine feature learning and pose optimization. Aoki et al. combine PointNet global features with an iterative pose optimization method . Wang et al. train graph neural network features by backpropagating through pose optimization.

We further advance this line of work. In particular, our Weighted Procrustes method reduces the complexity of optimization from quadratic to linear and enables the use of dense correspondences for highly accurate registration of real-world scans.

Deep Global Registration

3D reconstruction systems typically take a sequence of partial 3D scans as input and recover a complete 3D model of the scene. These partial scans are scene fragments, as shown in Fig. 1. In order to reconstruct the scene, reconstruction systems often begin by aligning pairs of fragments . This stage is known as pairwise registration. The accuracy and robustness of pairwise registration are critical and often determine the accuracy of the final reconstruction.

Our pairwise registration pipeline begins by extracting pointwise features. These are matched to form a set of putative correspondences. We then use a high-dimensional convolutional network (ConvNet) to estimate the veracity of each correspondence. Lastly, we use a Weighted Procrustes method to align 3D scans given correspondences with associated likelihood weights, and refine the result by optimizing a robust objective.

To prepare for registration, we extract pointwise features that summarize geometric context in the form of vectors in metric feature space. Our pipeline is compatible with many feature descriptors. We use Fully Convolutional Geometric Features (FCGF) , which have recently been shown to be both discriminative and fast. FCGF are also compact, with dimensionality as low as 16 to 32, which supports rapid neighbor search in feature space.

2 Correspondence Confidence Prediction

Note that the convolution operator is translation invariant, thus our 6D ConvNet will generate the same output regardless of the absolute position of inputs in 3D. We use a similar network architecture to Choy et al. to create a 6D convolutional network with skip connections within the spatial resolution across the network. The architecture of the 6D ConvNet is shown in Fig. 2. During training, we use the binary cross-entropy loss between the likelihood prediction that a correspondence $(i,j)$ is an inlier, $p_{(i,j)}\in$, and the ground-truth correspondences $\mathcal{P}$ to optimize the network parameters:

where $p^{C}=1-p$ and $|\mathcal{M}|$ is the cardinality of the set of putative correspondences.

3 Weighted Procrustes for SE​(3)SE3\text{SE}(3)

The inlier likelihood estimated by the 6D ConvNet provides a weight for each correspondence. The original Procrustes method minimizes the mean squared error between corresponding points $\frac{1}{N}\sum_{(i,j)\in\mathcal{M}}\|\mathbf{x}_{i}-\mathbf{y}_{j}\|^{2}$ and thus gives equal weight to all correspondences. In contrast, we minimize a weighted mean squared error $\sum_{(i,j)\in\mathcal{M}}w_{(i,j)}\|\mathbf{x}_{i}-\mathbf{y}_{j}\|^{2}$. This change allows us to pass gradients through the weights, rather than through the position , and enables the optimization to scale to dense correspondence sets.

Formally, Weighted Procrustes analysis minimizes:

The Weighted Procrustes method generates rotation $\hat{R}$ and translation $\hat{\mathbf{t}}$ as outputs that depend on the weight vector $\mathbf{w}$. In our current implementation, $\hat{R}$ and $\hat{\mathbf{t}}$ are directly sent to the robust registration module in Section 4 as an initial pose. However, we briefly demonstrate that they can also be embedded in an end-to-end registration pipeline, since Weighted Procrustes is differentiable. From a top-level loss function $L$ of $\hat{R}$ and $\hat{\mathbf{t}}$, we can pass the gradient through the closed-form solver, and update parameters in downstream modules:

where $L(\hat{R},\hat{\mathbf{t}})$ can be defined as the combination of differentiable rotation error (RE) and translation error (TE) between predictions $\hat{R},\hat{\mathbf{t}}$ and ground-truth $R^{*},\mathbf{t}^{*}$:

or the Forbenius norm of relative transformation matrices defined in . The final loss is the weighted sum of $L_{\text{rot}}$, $L_{\text{trans}}$, and $L_{\text{bce}}$.

Robust Registration

In this section, we propose a fine-tuning module that minimizes a robust loss function of choice to improve the registration accuracy. We use a gradient-based method to refine poses, where a continuous representation for rotations is adopted to remove discontinuities and construct a smooth optimization space. This module initializes the pose from the prediction of the Weighted Procrustes method. During iterative optimization, unlike Maken et al. , who find the nearest neighbor per point at each gradient step, we rely on the correspondence likelihoods from the 6D ConvNet, which is estimated only once per initialization.

In addition, our framework naturally offers a failure detection mechanism. In practice, Weighted Procrustes may generate numerically unstable solutions when the number of valid correspondences is insufficient due to small overlaps or noisy correspondences between input scans. By computing the ratio of the sum of the filtered weights to the total number of correspondences, i.e. $\sum_{i}\phi(w_{i})/|\mathcal{M}|$, we can easily approximate the fraction of valid correspondences and predict whether an alignment may be unstable. When this fraction is low, we resort to a more time-consuming but accurate registration algorithm such as RANSAC or a branch-and-bound method to find a numerically stable solution. In other words, we can detect when our system might fail before it returns a result and fall back to a more accurate but time-consuming algorithm, unlike previous end-to-end methods that use globally pooled latent features or a singly stochastic matrix – such latent representations are more difficult to interpret.

To initialize $\mathbf{a}_{1},\mathbf{a}_{2}$, we simply use the first two columns of the rotation matrix $R$, i.e., $\mathbf{b}_{1}$, $\mathbf{b}_{2}$. For convenience, we define $f^{-1}$ as $f^{-1}(f(R))=R$ though this inverse function is not unique as there are infinitely many choices of $\mathbf{a}_{1},\mathbf{a}_{2}$ that map to the same $R$.

2 Energy Minimization

We use a robust loss function to fine-tune the registration between predicted inlier correspondences. The general form of the energy function is

Experiments

We analyze the proposed model in two registration scenarios: pairwise registration where we estimate an $\text{SE}(3)$ transformation between two 3D scans or fragments, and multi-way registration which generates a final reconstruction and camera poses for all fragments that are globally consistent. Here, pairwise registration serves as a critical module in multi-way registration.

For pairwise registration, we use the 3DMatch benchmark which consists of 3D point cloud pairs from various real-world scenes with ground truth transformations estimated from RGB-D reconstruction pipelines . We follow the train/test split and the standard procedure to generate pairs with at least 30% overlap for training and testing . For multi-way registration, we use the simulated Augmented ICL-NUIM dataset for quantitative trajectory results, and Indoor LiDAR RGB-D dataset and Stanford RGB-D dataset for qualitative registration visualizations. Note in this experiment we use networks trained on the 3DMatch training set and do not fine-tune on the other datasets. This illustrates the generalization abilities of our models. Lastly, we use KITTI LIDAR scans for outdoor pairwise registration. As the official registration splits do not have labels for pairwise registration, we follow Choy et al. to create pairwise registration train/val/test splits.

For all indoor experiments, we use 5cm voxel downsampling , which randomly subsamples a single point within each 5cm voxel to generate point clouds with uniform density. For safeguard registration, we use RANSAC and the safeguard threshold $\tau_{s}=0.05$, which translates to 5% of the correspondences should be valid. We train learning-based state-of-the-art models and our network on the training split of the 3DMatch benchmark. During training, we augment data by applying random rotations varying from $-$180 to 180 degrees around a random axis. Ground-truth pointwise correspondences are found using nearest neighbor search in 3D space. We train the 6-dimensional ConvNet on a single Titan XP with batch size 4. SGD is used with an initial learning rate $10^{-1}$ and an exponential learning rate decay factor 0.99.

In this section, we report the registration results on the test set of the 3DMatch benchmark , which contains 8 different scenes as depicted in Fig. 3. We measure translation error (TE) defined in Eq. 6, rotation error (RE) defined in Eq. 7, and recall. Recall is the ratio of successful pairwise registrations and we define a registration to be successful if its rotation error and translation error are smaller than predefined thresholds. Average TE and RE are computed only on these successfully registered pairs since failed registrations return poses that can be drastically different from the ground truth, making the error metrics unreliable.

We compare our methods with various classical methods and state-of-the-art learning based methods . All the experiments are evaluated on an Intel i7-7700 CPU and a GTX 1080Ti graphics card except for Go-ICP tested on an Intel i7-5820K CPU. In Table 1, we measure recall with the TE threshold 30cm which is typical for indoor scene relocalization , and RE threshold 15 degrees which is practical for partially overlapping scans from our experiments. In Fig. 4, we plot the sensitivity of recall on both thresholds by changing one threshold and setting the other to infinity. Fig. 5 includes detailed statistics on separate test scenes. Our system outperforms all the baselines on recall by a large margin and achieves the lowest translation and rotation error consistently on most scenes.

Classical methods. To compare with classical methods, we evaluate point-to-point ICP, Point-to-plane ICP, RANSAC , and FGR , all implemented in Open3D . In addition, we test the open-source Python bindings of Go-ICP and Super4PCS . For RANSAC and FGR, we extract FPFH from voxel-downsampled point clouds. The results are shown in Table 1.

ICP variants mostly fail as the dataset contains challenging 3D scan sequences with small overlap and large camera viewpoint change. Super4PCS, a sampling-based algorithm, performs similarly to Go-ICP, an ICP variant with branch-and-bound search.

Feature-based methods, FGR and RANSAC, perform better. When aligning 5cm-voxel-downsampled point clouds, RANSAC achieves recall as high as 70%, while FGR reaches 40%. Table 1 also shows that increasing the number of RANSAC iterations by a factor of 2 only improves performance marginally. Note that our method is about twice as fast as RANSAC with 2M iterations while achieving higher recall and registration accuracy.

Learning-based methods. We use 3DRegNet , Deep Closest Point (DCP) , PRNet , and PointNetLK as our baselines. We train all the baselines on 3DMatch with the same setup and data augmentation as ours for all experiments.

For 3DRegNet, we follow the setup outlined in , except that we do not manually filter outliers with ground truth, and train and test with the standard realistic setup. We find that the registration loss of 3DRegNet does not converge during training and the rotation and translation errors are consistently above 30 degrees and 1m during test.

We train Deep Closest Point (DCP) with 1024 randomly sampled points for each point cloud for 150 epochs . We initialize the network with the pretrained weights provided by the authors. Although the training loss converges, DCP fails to achieve reasonable performance for point clouds with partial overlap. DCP uses a singly stochastic matrix to find correspondences, but this formulation assumes that all points in point cloud $X$ have at least one corresponding point in the convex hull of point cloud $Y$. This assumption fails when some points in $X$ have no corresponding points in $Y$, as is the case for partially overlapping fragments. We also tried to train PRNet on our setup, but failed to get reasonable results due to random crashes and high-variance training losses.

Lastly, we fine-tune PointNetLK on 3DMatch for 400 epochs, starting from the pretrained weights provided by the authors. PointNetLK uses a single feature that is globally pooled for each point cloud and regresses the relative pose between objects, and we suspect that a globally pooled feature fails to capture complex scenes such as 3DMatch.

In conclusion, while working well on object-centric synthetic datasets, current end-to-end registration approaches fail on real-world data. Unlike synthetic data, real 3D point cloud pairs contain multiple objects, partial scans, self-occlusion, substantial noise, and may have only a small degree of overlap between scans.

2 Multi-way Registration

Multi-way registration for RGB-D scans proceeds via multiple stages. First, the pipeline estimates the camera pose via off-the-shelf odometry and integrates multiple 3D scans to reduce noise and generate accurate 3D fragments of a scene. Next, a pairwise registration algorithm roughly aligns all fragments, followed by multi-way registration which optimizes fragment poses with robust pose graph optimization .

We use a popular open-source implementation of this registration pipeline and replace the pairwise registration stage in the pipeline with our proposed modules. Note that we use the networks trained on the 3DMatch training set and test on the multi-way registration datasets ; this demonstrates cross-dataset generalization.

We test the modified pipeline on the Augmented ICL-NUIM dataset for quantitative trajectory results, and Indoor LiDAR RGB-D dataset and Stanford RGB-D dataset for qualitative registration visualizations. We measure the absolute trajectory error (ATE) on the Augmented ICL-NUIM dataset with simulated depth noise. As shown in Table 2, compared to state-of-the-art online SLAM and offline reconstruction methods , our approach yields consistently low error across scenes.

For qualitative results, we compare pairwise fragment registration on these scenes against FGR and RANSAC in Fig. 6. Full scene reconstruction results are shown in the supplement.

3 Outdoor LIDAR Registration

We use outdoor LIDAR scans from the KITTI dataset for registration, following . The registration split of Choy et al. uses GPS-IMU to create pairs that are at least 10m apart and generated ground-truth transformation using GPS followed by ICP to fix errors in GPU readings. We use FCGF features trained on the training set of the registration split to find the correspondences and trained the 6D ConvNet for inlier confidence prediction similar to how we trained the system for indoor registration. We use voxel size 30cm for downsampling point clouds for all experiments. Registration results are reported in Tab. 3 and visualized in Fig. 7.

Conclusion

We presented Deep Global Registration, a learning-based framework that robustly and accurately aligns real-world 3D scans. To achieve this, we used a 6D convolutional network for inlier detection, a differentiable Weighted Procrustes algorithm for scalable registration, and a gradient-based optimizer for pose refinement. Experiments show that our approach outperforms both classical and learning-based registration methods, and can serve as a ready-to-use plugin to replace alternative registration methods in off-the-shelf scene reconstruction pipelines.

References