Learning multiview 3D point cloud registration

Introduction

Downstream tasks in 3D computer vision, such as semantic segmentation and object detection typically require a holistic representation of the scene. The capability of aligning and fusing individual point cloud fragments, which cover only small parts of the environment, into a globally consistent holistic representation is therefore essential and has several use cases in augmented reality and robotics. Pairwise registration of adjacent fragments is a well studied problem and traditional approaches based on geometric constraints and hand-engineered feature descriptors have shown successful results to some extent. Nevertheless, in the recent years, research on local descriptors for pairwise registration of 3D point clouds is centered on deep learning approaches that succeed in capturing and encoding evidence hidden to hand-engineered descriptors. Furthermore, novel end-to-end methods for pairwise point cloud registration were recently proposed . While demonstrating good performance for many tasks, pairwise registration of individual views of a scene has some conceptual drawbacks: (i) low overlap of adjacent point clouds can lead to inaccurate or wrong matches, (ii) point cloud registration has to rely on very local evidence, which can be harmful if 3D scene structure is scarce or repetitive, (iii) separate post-processing is required to combine all pair-wise matches into a global representation. Compared to the pairwise methods, globally consistent multiview alignment of unorganized point cloud fragments is yet to fully benefit from the recent advances achieved by the deep learning methods. State-of-the art methods typically still rely on a good initialization of the pairwise maps, which they try to refine globally in a subsequent decoupled step . A general drawback of this hierarchical procedure is that global noise distribution over all nodes of the pose graph ends up being far from random, i.e. significant biases persist due to the highly correlated initial pairwise maps.

In this paper, we present, to the best of our knowledge, the first end-to-end data driven multiview point cloud registration algorithm. Our method takes a set of potentially overlapping point clouds as input and outputs a global/absolute transformation matrix per each of the input scans (c.f. Fig. 1). We depart from a traditional two-stage approach where the individual stages are detached from each other and directly learn to register all views of a scene in a globally consistent manner.

We formulate the traditional two-stage approach in an end-to-end neural network, which in the forward pass solves two differentiable optimization problems: (i) the Procrustes problem for the estimation of the pairwise transformation parameters and (ii) the spectral relaxation of the transformation synchronization.

We propose a confidence estimation block that uses a novel overlap pooling layer to predict the confidence in the estimated pairwise transformation parameters.

We cast the mutliview 3D point cloud registration problem as an iterative reweighted least squares (IRLS) problem and iteratively refine both the pairwise and absolute transformation estimates.

Resulting from the aforementioned contributions, the proposed multiview registration algorithm (i) is very efficient to compute, (ii) achieves more accurate scan alignments because the residuals are being fed back to the pairwise network in an iterative manner, (iii) outperforms current state-of-the-art on pairwise as well as multiview point cloud registration.

Related Work

The traditional pairwise registration pipeline consists of two stages: the coarse alignment stage, which provides the initial estimate of the relative transformation parameters and the refinement stage that iteratively refines the transformation parameters by minimizing the 3D registration error under the assumption of rigid transformation.

The former is traditionally performed by using either handcrafted or learned 3D local features descriptors to establish the pointwise candidate correspondences in combination with a RANSAC-like robust estimator or geometric hashing . A parallel stream of works relies on establishing correspondences using the 4-point congruent sets. In the refinement stage, the coarse transformation parameters are often fine-tuned with a variant of the iterative closest point (ICP) algorithm . ICP-like algorithms perform optimization by alternatively hypothesizing the correspondence set and estimating the new set of transformation parameters. They are known to not be robust against outliers and to converge to a global optimum only when starting with a good prealingment . ICP algorithms are often extended to use additional radiometric, temporal or odometry constraints . Contemporary to our work, propose to integrate coarse and fine pairwise registration stages into an end-to-end learnable algorithm. Using a deep network, formulates the object tracking as a relative motion estimation of two point sets.

Multiview registration Multiview, global point cloud registration methods aim at resolving hard or ambiguous cases that arise in pairwise methods by incorporating cues from multiple views. The first family of methods employ a multiview ICP-like scheme to optimize for camera poses as well as 3D point correspondences . A majority of these suffer from increased complexity of correspondence estimation. To alleviate this, some approaches only optimize for motion and use the scans to evaluate the registration error . Taking a step further, other modern methods make use of the global cycle-consistency and optimize only over the poses starting from an initial set of pairwise maps. This efficient approach is known as synchronization . Global structure-from-motion aims to synchronize the observed relative motions by decomposing rotation, translation and scale components. proposes a global point cloud registration approach using two networks, one for pose estimation and another modelling the scene structure by estimating the occupancy status of global coordinates.

Probably the most similar work to ours is , where the authors aim to adapt the edge weights for the transformation synchronization layer by learning a data driven weighting function. A major conceptual difference to our approach is that relative transformation parameters are estimated using FPFH in combination with FGR and thus, unlike ours, are not learned. Furthermore, in each iteration has to convert the point clouds to depth images as the weighting function is approximated by a 2D CNN. On the other hand our whole approach operates directly on point clouds, is fully differentiable and therefore facilitates learning a global, multiview point cloud registration in an end-to-end manner.

End-to-End Multiview 3D Registration

In this section we derive the proposed multiview 3D registration algorithm as a composition of functions depending upon the data. The network architectures used to approximate these functions are then explained in detail in Sec 4. We begin with a new algorithm for learned pairwise point cloud registration, which uses two point clouds as input and outputs estimated transformation parameters (Sec. 3.1). This method is extended to multiple point clouds by using a transformation synchronization layer amenable to backpropagation (Sec. 3.2). The input graph to this synchronization layer encodes, along with the relative transformation parameters, the confidence in these pairwise maps, which is also estimated using a novel neural network, as edge information. Finally, we propose an IRLS scheme (Sec. 3.3) to refine the global registration of all point clouds by updating the edge weights as well as the pairwise poses.

In current state-of-the-art edges $\mathcal{E}$ of $\mathcal{G}$ are initialized with (noisy) relative transformation parameters $\{\mathbf{M}_{ij}\}$, obtained by an independent, auxiliary pairwise registration algorithm. Global scene consistency is enforced via a subsequent synchronization algorithm. In contrast, we propose a joint approach where pairwise registration and transformation synchronization are tightly coupled as one fully differentiable component, which leads to an end-to-end learnable, global registration pipeline.

where $\phi({\mathbf{p},\mathbf{Q}})$ is a correspondence function that maps the points $\mathbf{p}$ to their corresponding points in point cloud $\mathbf{Q}$. The formulation of Eq. 3 facilitates a differentiable closed-form solution, which is—subject to the noise distribution—close to the ground truth solution . However, least square solutions are not robust and thus Eq. 3 will yield wrong transformation parameters in case of high outlier ratio. In practice, the mapping $\phi({\mathbf{p},\mathbf{Q}})$ is far from ideal and erroneous correspondences typically dominate. To circumvent that, Eq. 3 can be robustified against outliers by introducing a heteroscedastic weighting matrix :

2 Differentiable transformation synchronization

Returning to the task of multiview registration, we again consider the initial set of point clouds $\mathcal{S}$. If no prior connectivity information is given, graph $\mathcal{G}$ can be initialized by forming ${N_{\mathcal{S}}\choose 2}$ point cloud pairs and estimating their relative transformation parameters as described in Sec. 3.1. The global transformation parameters can be estimated either jointly (transformation synchronization) or by dividing the problem into rotation synchronization and translation synchronization . Herein, we opt for the latter approach, which under the spectral relation admits a differentiable closed-form solution .

Rotation synchronization The goal of rotation synchronization is to retrieve global rotation matrices $\{\mathbf{R}_{i}^{*}\}$ by solving the following minimization problem based on their observed ratios $\{\hat{\mathbf{R}}_{ij}\}$

where the weigths $c_{ij}:=\zeta_{\text{init}}(\bm{\Gamma})$ represent the confidence in the relative transformation parameters $\hat{\mathbf{M}}_{ij}$. Under the spectral relaxation Eq. 5 admits a closed-form solution, which is provided in the supp. material .

Similarly, the goal of translation synchronization is to retrieve global translation vectors $\{\mathbf{t}_{i}^{*}\}$ that minimize the following least squares problem

The differentiable closed-form solution to Eq. 6 is again provided in the supp. material.

3 Iterative refinement of the registration

The above formulation (Sec. 3.1 and 3.2) facilitates an implementation in an iterative scheme, which in turn can be viewed as an IRLS algorithm. We can start each subsequent iteration $(k+1)$ by pre-aligning the point cloud pairs using the synchronized estimate of the relative transformation parameters $\mathbf{M}^{*(k)}_{ij}=\mathbf{M}_{i}^{*(k)}\mathbf{M}_{j}^{*(k)^{-1}}$ from iteration $(k)$ such that $\mathbf{Q}^{(k+1)}:=\mathbf{M}^{*(k)}_{ij}\otimes\mathbf{Q}$, where $\otimes$ denotes applying the transformation $\mathbf{M}^{*(k)}_{ij}$ to point cloud $\mathbf{Q}$. Additionally, weights $\mathbf{w}^{(k)}$ and residuals $\mathbf{r}^{(k)}$ of the previous iteration can be used as a side information in the correspondence weighting function. Therefore, $\psi_{\text{init}}(\cdot)$ is extended to

Network Architecture

We implement our proposed multiview registration algorithm as a deep neural network (Fig. 2). In this section, we first describe the architectures used to aproximate $\phi(\cdot),\psi_{\text{init}}(\cdot),\psi_{\text{iter}}(\cdot),\zeta_{\text{init}}(\cdot)$ and $\zeta_{\text{iter}}(\cdot)$, before integrating them into one fully differentiable, end-to-end trainable algorithm.

Our approximation of the correspondence function $\phi(\cdot)$ extends a recently proposed fully convolutional 3D feature descriptor FCGF with a soft assignment layer. FCGF operates on sparse tensors and computes $32$ dimensional descriptors for each point of the sparse point cloud in a single pass. Note that the function $\phi(\cdot)$ could be approximated with any of the recently proposed learned feature descriptors , but we choose FCGF due to its high accuracy and low computational complexity.

Let $\mathbf{F}_{\mathbf{P}}$ and $\mathbf{F}_{\mathbf{Q}}$ denote the FCGF embeddings of point clouds $\mathbf{P}$ and $\mathbf{Q}$ obtained using the same network weights, respectively. Pointwise correspondences $\{\phi(\cdot)\}$ can then be established by a nearest neighbor (NN) search in this high dimensional feature space. However, the selection rule of such hard assignments is not differentiable. We therefore form the NN-selection rule in a probabilistic manner by computing a probability vector $\mathbf{s}$ of the categorical distribution . The stochastic correspondence of the point $\mathbf{p}$ in the point cloud $\mathbf{Q}$ is then defined as

where $d_{l}:=||\mathbf{f}_{\mathbf{p}}-(\mathbf{F}_{\mathbf{Q}})_{l}||_{2}$, $\mathbf{f}_{\mathbf{p}}$ is the FCGF embedding of the point $\mathbf{p}$ and $t$ denotes the temperature parameter. In the limit $t\to 0$ the $\phi(\mathbf{p},\mathbf{Q})$ converges to the deterministic NN-search .

We follow and supervise the learning of $\phi(\cdot)$ with a correspondence loss $\mathcal{L}_{c}$, which is defined as the hardest contrastive loss and operates on the FCGF embeddings

where $\mathcal{P}$ is a set of all the positive pairs in a FCGF mini batch $N_{\text{FCGF}}$ and $\mathcal{N}$ is a random subset of all features that is used for the hardest negative mining. $m_{p}=0.1$ and $m_{n}=1.4$ are the margins for positive and negative pairs respectively. The detailed network architecture of $\phi(\cdot)$ as well as the training configuration and parameters are available in the supp. material.

Despite the good performance of the FCGF descriptor, several putative correspondences $\bm{\Gamma}^{\prime}\subset\bm{\Gamma}$ will be false. Furthermore, the distribution of inliers and outliers does not resemble noise but rather shows regularity . We thus aim to learn this regularity from the data using a deep neural network. Recently, several networks representing a complex weighting function for filtering of 2D or 3D feature correspondences have been proposed.

loss, where $\mathcal{L}_{\text{class}}$ denotes the binary cross entropy loss and

is used to penalize the deviation from the ground truth transformation parameters $\mathbf{M}^{\text{GT}}_{ij}$. $\alpha_{\text{reg}}$ and $\beta_{\text{reg}}$ are used to control the contribution of the individual loss functions.

Along with the estimated relative transformation parameters $\hat{\mathbf{M}_{ij}}$, the edges of the graph $\mathcal{G}$ encode the confidence $c_{ij}$ in those estimates. Confidence encoded in each edge of the graph consist of (i) the local confidence $c_{ij}^{\text{local}}$ of the pairwise transformation estimation and (ii) the global confidence $c_{ij}^{\text{global}}$ derived from the transformation synchronization. We formulate the estimation of $c_{ij}^{\text{local}}$ as a classification task and argue that some of the required information is encompassed in the features of the second-to-last layer of the registration block. Let $\mathbf{X}^{\text{conf}}_{ij}=f_{\theta}^{(-2)}(\cdot)$ denote the output of the second-to-last layer of the registration block, we propose an overlap pooling layer $f_{\text{overlap}}$ that extracts a global feature $\mathbf{x}^{\text{conf}}_{ij}$ by performing the weighted average pooling as

The obtained global feature is concatenated with the ratio of inliers $\delta_{ij}$ (i.e., the number of correspondences whose weights are higher than a given threshold) and fed to the confidence estimation network with three fully connected layers $(129-64-32-1)$, followed by a ReLU activation function. The local confidence can thus be expressed as

The $\zeta_{\text{init}}(\cdot)$ incorporates the local confidence in the relative transformation parameters. On the other hand, the output of the transformation synchronization layer provides the information how the input relative transformations agree globally with the other edges. In fact, traditional synchronization algorithms only use this global information to perform the reweighting of the edges in the iterative solutions, because they do not have access to the local confidence information. Global confidence in the relative transformation parameters $c_{ij}^{\text{global}}$ can be expressed with the Cauchy weighting function

where $r^{*}_{ij}=||\hat{\mathbf{M}}_{ij}-\mathbf{M}_{i}^{*}\mathbf{M}_{j}^{*^{T}}||_{F}$ and following $b=1.482\mkern 3.0mu\gamma\mkern 3.0mu\operatorname{med}(|\mathbf{r}^{*}-\operatorname{med}(\mathbf{r}^{*})|)$ with $\operatorname{med}(\cdot)$ denoting the median operator and $\mathbf{r}^{*}$ the vectorization of residuals $r^{*}_{ij}$. Since local and global confidence provide complementary information about the relative transformation parameters, we combine them into a joined confidence $c_{ij}$ using their harmonic mean:

where the $\beta$ balances the contribution of the local and global confidence estimates and is learned during training.

The individual parts of the network are connected into an end-to-end multiview 3D registration algorithm as shown in Fig. 2The network is implemented in Pytorch . A pseudo-code of the proposed approach is provided in the supp. material.. We pre-train the individual sub-networks (training details available in the supp. material) before fine-tuning the whole model in an end-to-end manner on the 3DMatch data set using the official train/test data split. In fine-tuning we use $N_{\text{FCGF}}=4$ to extract the FCGF features and randomly sample feature vectors of 2048 points per fragment. These features are used in the soft assignment (softNN) to form the putative correspondences of $N_{\mathcal{S}}\choose 2$ point clouds pairsWe assume a fully connected graph during training but are able to consider the connectivity information, if provided., which are fed to the pairwise registration network. The output of the pairwise registration is used to build the graph, which is input to the transformation synchronization layer. The iterative refinement of the transformation parameters is performed four times. We supervise the fine tuning using the joint multiview registration loss

where the transformation synchronization $\mathcal{L}_{sync}$ loss reads

We fine-tune the whole network for $2400$ iterations using Adam optimizer with a learning rate of $5\times 10^{-6}$.

Experiments

We conduct the evaluation of our approach on the publicly available benchmark datasets 3DMatch , Redwood and ScanNet . First, we evaluate the performance, efficiency, and the generalization capacity of the proposed pairwise registration algorithm on 3DMatch and Redwood dataset respectively (Sec. 5.1). We then evaluate the whole pipeline on the global registration of the point cloud fragments generated from RGB-D images, which are part of the ScanNet dataset .

We begin by evaluating the pairwise registration part of our algorithm on a traditional geometric registration task. We compare the results of our method to the state-of-the-art data-driven feature descriptors 3DMatch , CGF , PPFNet , 3DSmoothNet (3DS) , and FCGF , which is also used as part of our algorithm, as well as to a recent network based registration algorithm 3DR . Following the evaluation procedure of 3DMatch we complement all the descriptor based methods with the RANSAC-based transformation parameter estimation. For our approach we report the results after the pairwise registration network (1-iter in Tab. 1) as well as the the output of the $\psi_{\text{iter}}(\cdot)$ in the $4^{\text{th}}$ iteration (4-iter in Tab. 1). The latter is already informed with the global information and serves primarily as verification that with the iterations our input to the Transf-Sync layer improves. Consistent with the 3DMatch evaluation procedure, we report the average recall per scene as well as for the whole dataset in Tab. 1.

The registration results show that our approach reaches the highest recall among all the evaluated methods. More importantly, it indicates that using the same features (FCGF), our method can outperform RANSAC-based estimation of the transformation parameters, while having a much lower time complexity (Tab. 2). The comparison of the results of 1-iter and 4-iter also confirms the intuition that feeding the residuals and weights of the previous estimation back to the pairwise registration block helps refining the estimated pairwise transformation parameters.

Generalization to other domains In order to test if our pairwise registration model can generalize to new datasets and unseen domains, we perform a generalization evaluation on a synthetic indoor dataset Redwood indoor . We follow the evaluation protocol of and report the average registration recall and precision across all four scenes. We compare our approach to the recent data driven approaches 3DMatch , CGF +FGR or CZK , RelativeNet (RN) , 3DR and traditional methods CZK and Latent RANSAC (LR) . Fig. 3 shows that our approach can achieve $\approx 4$ percentage points higher recall than state-of-the-art without being trained on synthetic data and thus confirming the good generalization capacity of our approach. Note that while the average precision across the scenes is low for all the methods, several works show that the precision can easily be increased using pruning without almost any loss in the recall.

2 Multiview registration performance

We finally evaluate the performance of our complete method on the task of multiview registration using the ScanNet dataset. ScanNet is a large RGBD dataset of indoor scenes. It provides the reconstructions, ground truth camera poses and semantic segmentations for $1513$ scenes. To ensure a fair comparison, we follow and use the same $32$ randomly sampled scenes for evaluation. For each scene we randomly sample $30$ RGBD images that are 20 frames apart and convert them to point clouds. The temporal sequence of the frames is discarded. In combination with the large temporal gap between the frames, this makes the test setting extremely challenging. Different to , we do not train our network on ScanNet, but rather perform direct generalization of the network trained on the 3DMatch dataset.

Evaluation protocol We use the standard evaluation protocol and report the empirical cumulative distribution function (ECDF) for the angular $a_{e}$ and translation $t_{e}$ deviations defined as

The ground truth rotations $\mathbf{R^{\text{GT}}}$ and translations $\mathbf{t^{\text{GT}}}$ are provided by the authors of ScanNet . In Tab. 3 we report the results for three different scenarios. ”FGR (Good)” and ”Ours (Good)” denote the scenarios in which we follow and use the computed pairwise registrations to prune the edges before the transformation synchronization if the median point distance in the overlappingThe overlapping regions are defined as parts, where after transformation, the points are less than $0.2$m away from the other point cloud. region after the transformation is larger than $0.1$m (FGR) or $0.05$m (ours). The EIGSE3 in ”Ours (Good)” is initialized using our pairwise estimates. On the other hand, ”all” denotes the scenario in which all $N_{\mathcal{S}}\choose 2$ pairs are used to build the graph. In all scenarios we prune the edges of the graph if the confidence estimation in the relative transformation parameters of that edge $c^{\text{local}}_{ij}$ drops below $\tau_{p}=0.85$. This threshold was determined on 3DMatch dataset and its effect on the performance of our approach is analyzed in detail in the supp. material. If during the iterations the pruning of the edges yields a disconnected graph we simply report the last valid values for each node before the graph becomes disconnected. A more sophisticated handling of the edge pruning and disconnected graphs is left for future work.

Analysis of the results As shown in Tab. 3 our approach can achieve a large improvement on the multiview registration tasks when compared to the baselines. Not only are the initial pairwise relative transformation parameters estimated using our approach more accurate than the ones of FGR , but they can also be further improved in the subsequent iterations. This clearly confirms the benefit of the feed-back loop of our algorithm. Furthermore even when directly considering all input edges our approach still proves dominant, even when considering the results of the scenario ”Good” for our competitors. More qualitative results of the multiview registration evaluation, including the failure cases, are available in the supp. material.

Computational complexity Low computational costs of pairwise and multiview registration is important for various fields like augmented reality or robotics. We first compare computation time of our pairwise registration component to RANSAC. In Tab. 2 we report the average time needed to register one fragment pair of the 3DMatch dataset as well as one whole scene. All timings were performed on a standalone computer with Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz, GeForce GTX 1080, and 32 GB RAM. Average time of performing softNN for a fragment pair is about $0.1\text{s}$, which is a approxiately four times faster than traditional nearest neighbor search (implemented using scikit-learn ). An even larger speedup (about $23$ times) is gained in the model estimation stage, where our approach requires a single forward pass (constant time) compared to up to 50000 iterations of RANSAC when the inlier ratio is $5\%$ and the desired confidence $0.99$We use the CPU-based RANSAC implementation that is provided in the original evaluation code of 3DMatch dataset .. This results in an overall run-time of about $80\text{s}$ for our entire multiview approach (including the feature extraction and transformation synchronization) for the Kitchen scene with 1770 fragment pairs. In contrast, feature extraction and pairwise estimation of transformation parameters with RANSAC takes $>1100$s. This clearly shows the efficiency of our method, being $>13$ times faster to compute (for a scene with $60$ fragments).

3 Ablation study

To get a better intuition how much the individual novelties in our approach contribute to the final performance, we carry out an ablation study on the ScanNet dataset. In particular, we analyze the proposed edge pruning scheme based on the confidence estimation block and Cauchy function as well as the impact of the iterative refinement of the relative transformation parameters.Additional results of the ablation study are included in the supp. material. The results of the ablation study are presented in Fig. 4.

We motivate the iterative refinement of the transformation parameters that are input to the Transf-Sync layer with a notion that the weights and residuals provide additional ques for their estimation. Results in Fig. 4 confirm this assumption. The input relative parameters in the 4-th iteration are approximately $2$ percentage points better that the initial estimate. On the other hand, Fig. 4 shows that at the high presence of outliers or inefficient edge pruning (see e.g., the results w/o edge pruning) the weights and the residuals actually provide a negative bias and worsen the results.

Edge pruning scheme There are several possible ways to implement the pruning of the presumable outlier edges. In our experiments we prune the edges based on the output of the confidence estimation block (w-conf.). Other options are to realize this step using the global confidence, i.e. the Cauchy weights defined in (14) (w-Cau.) or not performing this at all (w/o). Fig. 4 clearly shows the advantage of using our confidence estimation block (gain of more than $20$ percentage points). Even more, due to preserving a large amount of outliers, alternative approaches preform even worse than the pairwise registration.

Conclusions

We have introduced an end-to-end learnable, multiview point cloud registration algorithm. Our method departs from the common two-stage approach and directly learns to register all views in a globally consistent manner. We augment the 3D descriptor FCGF by a soft correspondence layer that pairs all the scans to compute initial matches, which are fed to a differentiable pairwise registration block resulting in transformation parameters as well as weights. A pose graph is constructed and a novel, differentiable iterative transformation synchronization layer globally refines weights and transformations. Experimental evaluation on common benchmark datasets show that our method outperforms state-of-the-art by more than 25 percentage points on average regarding the rotation error statistics. Moreover, our approach is $>13$ times faster than RANSAC-based methods in a multiview setting of 60 scans, and generalizes better to new scenes ($\approx 4$ percentage points higher recall on Redwood indoor compared to state-of-the-art).

Acknowledgements. This work is partially supported by Stanford-Ford Alliance, NSF grant IIS-1763268, Vannevar Bush Faculty Fellowship, Samsung GRO program and the Stanford SAIL Toyota Research Center. We thank NVIDIA Corp. for providing the GPUs used in this work.

References

A Supplementary Material

B Supplementary material

In this supplementary material, we provide additional information about the proposed algorithm (Sec. B.1-B.2 and Alg. 1), network architectures and training configurations (Sec. B.3), an extended ablation study (Sec. B.5) as well as additional visualizations (Sec. B.7). The source code and pretrained models are publicly available under https://github.com/zgojcic/3D_multiview_reg.

For the sake of completeness we summarize the closed-form differentiable solution of the weighted least square pairwise registration problem

Let $\overline{\mathbf{p}}$ and $\overline{\mathbf{q}}$

where $\mathbf{W}=\text{diag}(w_{1},\ldots,w_{N})$ . Considering the singular value decomposition $\mathbf{S}=\mathbf{U}\mathbf{\Sigma}\mathbf{V}^{T}$ the solution to Eq. 22 is given by

where $\text{det}(\cdot)$ denotes computing the determinant and is used here to avoid creating a reflection matrix. Finally, $\hat{\mathbf{t}}_{ij}$ is computed as

B.2 Closed-form solution of Eq. 5 and 6

In this section we summarize the closed form solutions to Eq. 5 and 6 from the main paper describing the rotation and translation synchronization, respectively.

The least squares formulation of the rotation synchronization problem

where $\mathbf{D}$ is the weighted degree matrix constructed as

and $\mathbf{A}$ is a block matrix of the relative rotations

Similarly, the closed-form solution to the least squares formulation of the translation synchronization

where $\mathcal{N}(i)$ denotes all the neighboring vertices of $\mathbf{S}_{i}$ in graph $\mathcal{G}$.

B.3 Network architecture and training details

This section describes the network architecture as well as the training details of the FCGF feature descriptor (Sec. B.3.1) and the proposed registration block (Sec. LABEL:sec:reg_block). Both networks are implemented in Pytorch and pretrained using the 3DMatch dataset .

The FCGF feature descriptor operates on sparse tensors that represent a point cloud in form of a set of unique coordinates $\mathbf{C}$ and their associated features $\mathbf{F}$

where $x_{i},y_{i},z_{i}$ are the coordinates of the $i$-th point in the point cloud and $f_{i}$ is the associated feature (in our case simply 1). FCGF is implemented using the Minkowski Engine, an auto-differentiation library, which provides support for sparse convolutions and implements all essential deep learning layers . We adopt the original, fully convolutional network design of FCGF that is depicted in Fig. 5. It has a UNet structure and utilizes skip connections and ResNet blocks to extract the per-point $32$ dim feature descriptors. To obtain the unique coordinates $\mathbf{C}$, we use a GPU implementation of the voxel grid downsampling with the voxel size $v:=2.5\,\text{cm}$.

We again follow and pre-train FCGF for 100 epochs using the point cloud fragments from the 3DMatch dataset . We optimize the parameters of the network using stohastic gradient descent with a batch size $4$ and an initial learning rate of $0.1$ combined with an exponential decay with $\gamma=0.99$. To introduce rotation invariance of the descriptors we perform a data augmentation by randomly rotating each of the fragments along an arbitrary direction, by a different rotation, sampled from the $[0^{\circ},360^{\circ})$ interval. The sampling of the positive and negative examples follows the procedure proposed in .

B.3.2 Registration block

The architecture of the registration block (same for $\psi_{\text{init}}(\cdot)$ and $\psi_{\text{iter}}(\cdot)$)For $\psi_{\text{init}}(\cdot)$ the input dimension is increased from 6 to 8 (weights and residuals added). follows and is based on the PointNet-like architecture where each of the fully connected layers ($\mathbf{P}$ in Fig. 6) operates on individual correspondences. The local context is then aggregated using the instance normalization layers defined as

where $N_{c}$ is the number of correspondences and $M_{c}$ is the number of clusters. $\mathbf{X}_{k}$ and $\mathbf{X}_{k+1}$ are the features at the level $k$ (before clustering) and $k+1$ (after clustering), respectively (see Fig. 6). Finally, $\mathbf{X}^{\prime}_{k+1}$ denotes the output of the last layer in the level $k+1$.

We pre-train the registration blocks using the same fragments from the 3DMatch dataset. Specifically, we first infer the FCGF descriptors and randomly sample $N_{c}=5000$ descriptors per fragment. We use these descriptors to compute the putative correspondences for all fragment pairs $(i,j)$ such that $i\leq j$. Based on the ground truth transformation parameters, we label these correspondences as inliers if the Euclidean distance between the points after the transformation is smaller than $7.5$ cm. At the start of the training (first 15000 iterations) we supervise the learning using only the binary cross-entropy loss. Once a meaningful number of correspondences can already be classified correctly we add the transformation loss. We train the network for $500$k iterations using Adam optimizer with the initial learning rate of $0.001$. We decay the learning rate every $1000$ iterations by multiplying it with $0.999$. To learn the rotation invariance we perform data augmentation, starting from the $25000$th iteration, by randomly sampling an angle from the interval $[0^{\circ},n_{a}\cdot 20^{\circ})$ where $n_{a}$ is initialized with zero and is then increased by $1$ every $5000$ iteration until the interval becomes $[0^{\circ},360^{\circ})$.

B.4 Pseudo-code

Alg. 1 shows the pseudo-code of our proposed approach. We iterate $k=4$ times over the network and transformation synchronization (i.e. Transf-Sync) layers and in each of those iterations we execute the Transf-Sync layer four times. Our implementation is constructed in a modular way (each part can be run on its own) and can accept a varying number of input point clouds with or without the connectivity information.

B.5 Extended ablation study

We extend the ablation study presented in the main paper, by analyzing the impact of edge pruning based on the local confidence (i.e. the output of the confidence estimation block) (Sec. B.5.1) and of the weighting scheme (Sec. B.6) on the angular and translation errors. The ablation study is performed on the point cloud fragments of the ScanNet dataset .

Results depicted in Fig. 7 show that the threshold value used for edge pruning has little impact on the angular and translation errors as long as it is larger than 0.2.

B.6 Impact of the harmonic mean weighting scheme

In this work, we have introduced a scheme for combining the local and global confidence using the harmonic mean (HM). In the following, we perform the analysis of this proposal and compare its performance to established methods based only on global information . To this end, we again consider the scenario ”Ours (Good)” as the input graph connectivity information. We compare the results of the proposed scheme (HM) to SE3 EIG , which proposes using the Cauchy function for computing the global edge confidence . Note, we use the same pairwise transformation parameters, estimated using the method proposed herein, for all methods.

It turns out that combining the local and global evidence about the graph connectivity is essential to achieve good performance. In fact, merely relying on local confidence estimates without HM weighting (denoted as ours; green) in Fig. 8) the Transf-Sync is unable to recover global transformations from the given graph connectivity evidence that is very noisy. Introducing the HM weighting scheme allows us to reduce the impact of noisy graph connectivity built solely using local confidence and can significantly improve performance after Transf-Sync block, which in turn enables us to outperform the SE3 EIG.

Fig. 9 shows that pruning the edges can help coping with noisy input graph connectivity built from the pairwise input. In principal, suppression of the edges with low confidence results in discarding the outliers that corrupt the l2 solution and as a result improves the performance of the Transf-Sync block.

B.7 Qualitative results

We provide some additional qualitative results in form of success and failure cases on selected scenes of 3DMatch (Fig. 10 and 11) and ScanNet (Fig. 12 and 13) datasets. Specifically, we compare the results of our whole pipeline Ours (After Sync.) to the results of SE3 EIG , pairwise registration results of our method from the first iteration Ours ($1^{st}$ iter.), and pairwise registration results of our method from the fourth iteration Ours ($4^{th}$ iter.). Both global methods (Ours (After Sync.) and SE3 EIG) use transformation parameters estimated by our proposed pairwise registration algorithm as input to the transformation synchronization. The failure cases of our method predominantly occur on point clouds with low level of structure (planar areas in Fig. 11 bottom) or high level of symmetry and repetitive structures (Fig. 13 top and bottom, respectively).