Interpolated Convolutional Networks for 3D Point Cloud Understanding

Introduction

Point cloud is an important data format obtained by 3D sensors and has shown extensive usage in many real-world tasks including autonomous driving , robotics , etc. Efficient learning from point cloud data remains a challenge to the research community, given the fact that point clouds are usually irregular, unordered and sparse.

In view of the great success of convolutional neural networks (CNNs) on 2D images, many endeavors have been made to adapt the convolution operation to 3D point clouds. Currently there are two main approaches to tackle this problem. The first type of attempts is to directly rasterize irregular point clouds into regular voxel grids, and adopts standard 3D convolutions to learn shape features. However, the transformation of irregular inputs leads to a loss of geometric information, and convolutions on dense voxel grids lead to heavy computational burden. Other approaches build local graphs in the neighborhood of each point in the Euclidean or feature space, then a continuous convolutional kernel is applied on each edge of the graph to learn geometric features. Continuous kernels are commonly modeled by Multi-layer Perceptrons (MLPs). These graph-based methods are able to directly process irregular data structure but have some drawbacks. The construction of local graphs is not sparsity invariant. Namely, different point cloud densities sampled from the same object surface lead to different neighborhood selections, thus it may produce various graph construction results. Besides, compared with discrete convolutions, utilizing MLPs to learn an arbitrary continuous function does not work well in practice .

In this paper, we propose a novel Interpolated Convolution operation (InterpConv) to address the existing problems in graph and 3D convolutional neural networks. Key to our approach is the use of discrete convolutional kernels and an interpolation function to explicitly measure geometric relations between input point clouds and kernel-weight coordinates. Unlike 3D convolutions which have to transform inputs into regular grids, our InterpConv directly takes irregular point clouds as inputs. Each $n\times n\times c$ convolutional kernel is split into $n^{2}$ kernel weights, each of which has a $1\times c$ weight vector and its own coordinate $p^{\prime}$ relative to the kernel center. The center of discrete convolutional kernels can be placed at any location in the 3D space and then the kernel-weight absolute coordinates can be determined for each kernel. Input points are interpolated to neighboring kernel-weight coordinates by an interpolation function. To guarantee InterpConv to be sparsity invariant, normalization on points is adopted in the neighborhood of each kernel weight vector. Finally, weighted convolutions can be calculated between kernel weight vectors and point cloud features that are associated to them. With spatially-discrete convolutional kernel weights and an explicitly defined interpolation function, our approach performs better than graph-based methods, which use continuous functions as convolutional kernels and implicitly learn geometric relations. See Figure 1 for illustration.

We further propose Interpolated Convolutional Neural Networks (InterpCNNs) based on InterpConvs. The classification network is composed of multi-layer and multi-receptive-field InterpConv blocks, which can capture both fine-grained geometric structures and context information. The segmentation network explores a deeper architecture to predict semantic labels for all input points. We evaluate our networks on several benchmark datasets, including ModelNet40 , ShapeNet Parts and S3DIS . Experiments show that our approach achieves state-of-the-art performances on those datasets.

The key contributions of our work are as follows:

We propose a novel Interpolated Convolution operation (InterpConv) to effectively deal with point cloud recognition problems. Such an operation is permutation and sparsity invariant, and can directly handle irregular point clouds;

We design Interpolated Convolutional Neural Networks based on InterpConvs. The networks perform better than Graph Neural Networks (GNNs) and 3D Convolutional Neural Networks (3D ConvNets) on point cloud recognition and segmentation problems.

Related Work

Our approach is closely related to other deep learning methods on point clouds. We introduce literatures on point cloud feature learning by regular grids and irregular inputs.

Learning from point clouds by regular grids. When facing irregular point clouds as inputs, an intuitive way is to transform this irregular data structure into regular grids. Some approaches transform 3D objects or point clouds into 2D regular grids, namely, images by multi-view projection, then 2D CNNs are utilized to learn from these images. Those methods work well owing to the great success of 2D CNNs on images. However, not all the geometric information is kept during projection, and those approaches are usually inefficient and time-consuming when handling sparse point cloud data.

An alternative approach is to rasterize point clouds into 3D regular grids. VoxNet transforms original point cloud data into occupancy grids, which store binary values to indicate whether the spaces are occupied. Then a 3D CNN is applied to learn from these voxel grids. The rasterization process loses some fine-grained geometric features, and 3D convolutions are both time and memory consuming. OctNet exploits the sparsity of voxel grids and uses unbalanced octrees to hierarchically partition the space, which saves much memory. Some other efforts have also been made to ease the computational burden but still cannot solve the loss of geometric information during rasterization.

Compared with the above-mentioned approaches, our approach directly takes irregular point clouds as inputs without rasterization, which is time-saving and accurate.

Learning from point clouds by irregular inputs. Recently there are many works trying to directly process irregular point cloud data. Pioneering work PointNet utilizes shared MLPs and a maxpooling layer, which is permutation invariant, to tackle unordered inputs and learn a global representation. PointNet++ exploits local structures by grouping and sampling point clouds, then a PointNet is applied in each group to aggregate local features. However, how to effectively partition and select point clouds remain a challenge. Many approaches explore new grouping and sampling strategies. In , new modules are added to original PointNet++ in order to gain a better performance.

Graph Neural Networks (GNNs) have been widely used to deal with irregular data structure. There are also a bunch of works trying to apply GNNs to solve point cloud processing problem. Those approaches usually build local graphs in the neighborhood of the Euclidean or feature space, utilize MLPs as continuous convolutional kernel functions, and aggregate local features by weighted sum or pooling from neighborhood to center. DGCNN proposes an EdgeConv operation which concatenates central and neighboring point features and learns new features by MLP and maxpooling. 3DGNN applies gated graph neural networks on semantic segmentation task. SpiderCNN defines a continuous kernel function as a product of step function and a Taylor polynomial. KCNet proposes a kernel correlation and graph pooling layer to exploit local structures. PointCNN applies $\chi$-transform operator on local graphs.

GNN-based methods still have some problems. First, the graph construction process based on K-nearest neighbors (KNN) is sensitive to point cloud density. Second, using MLPs to directly learn from point coordinates is inefficient, as it ignores some explicitly defined geometric relations. Different from those methods, our approach is not sensitive to point cloud density due to the proposed normalization term, and geometric relations between discrete kernel weights and point clouds are explicitly defined by an interpolation function.

Method

In this section, we first revisit convolutions on different types of point sets. We then introduce our proposed Interpolated Convolution operation (InterpConv) and key elements of the InterpConv algorithm. Finally, we show details for our network architectures for 3D object recognition and semantic segmentation.

Standard 2D and 3D convolutions have achieved great success in handling regularly-arranged data such as images and voxel grids. When it comes to sparse and irregular point sets such as 3D point clouds, multiple variants of convolutions have been proposed. In this section, we review those convolutions to motivate the design of our InterpConv operation.

It is worth noting that applying graph neural networks to handle point clouds essentially shares the same idea with continuous convolutions.

Replacing discrete kernel weights $W(p^{\prime})$ with continuous functions $W(p_{\delta})$ remains some problems. Simply learning continuous functions by MLPs cannot always work in practice . The predicted parameters might be too many, and the learning process is inefficient and sometimes unstable. Knowledge on the great success of discrete kernels in images cannot be transfered to point clouds recognition tasks as well.

2 Interpolated Convolution for 3D Point Clouds

We note that unlike standard convolutions, where kernel weights are regularly-arranged, kernel-weight coordinates $p^{\prime}$ in InterpConvs can be set flexibly or even learned during training.

There are three key parts of our proposed InterpConv operation: spatially-discrete kernel weights $W$, an interpolation function $T$, and a normalization term $N$. We first discuss those three parts separately, and then introduce the complete algorithm.

Discrete kernel weights. In 2D convolution , one kernel can be represented as an $n\times n\times c$ tensor, where $n$ denotes the kernel size and $c$ denotes the number of channels. In , one kernel is split into $n\times n$ weight vectors, each of which is of size $1\times c$. By doing so, kernel weights no longer have to be regularly-arranged, but can be flexibly placed on 2D grids.

In our approach, we further improve this idea by defining a set of kernel weight vectors for each convolutional kernel in the 3D Euclidean space. Each kernel weight vector $W(p^{\prime})$ has a 3D coordinate $p^{\prime}$ to store its location relative to the kernel center, and its weights are stored in a $1\times c$ vector, which will be initialized and updated during training. The vector coordinate $p^{\prime}$ can either be fixed or updated during training. To simplify the problem, we fix kernel-weight coordinates in most experiments and organize them as a cube, namely, kernel weight vectors are arranged at $3\times 3\times 3$ 3D regular grids if the total number of kernel weight vectors is $27$. We note that this is an analogy of standard $3\times 3\times 3$ discrete convolutions while kernel weight vectors can theoretically be placed at arbitrary locations in the 3D space.

As we arrange kernel weight vectors as a cube, we define two important hyperparameters: the kernel size $n\times n\times n$ and kernel length $l$. The coordinate set of spatially-discrete kernel weight vectors can be formulated as

Interpolation functions. One problem to apply discrete kernels on irregular point clouds is that kernel weight vectors’ spatial locations generally do not align with input points. Naively rasterizing point clouds into regular grids solves part of the problem, but at the cost of losing local structures. In our approach, we solve this problem while keep all fine-grained structures by adopting an interpolation function. Namely, we first find a set of input points near each kernel weight vector, and then interpolate their features to be assigned to the kernel weight vectors for convolution. We propose two interpolation functions: trilinear interpolation and Gaussian interpolation.

Trilinear interpolation is a commonly-used method to approximate the value of an intermediate point in a 3D grid by values of adjacent lattice points. The intermediate point’s value is calculated by a weighted sum of lattice points’ values, and the weights characterize closeness between intermediate and lattice points. In our approach, we adopt the inverse process of trilinear interpolation. Namely, we first compute the weights that lattice points (kernel-weight coordinates) contribute to the intermediate point (input point) and then we inversely assign the input point feature to adjacent kernel-weight coordinates with those weights.

For trilinear interpolation, we find $8$ adjacent kernel-weight coordinates $p^{\prime}$ for each input point $p_{\delta}$ in the kernel, and then we normalize input point and kernel weights into a unit-length cube. Finally we compute the trilinear interpolation weights by

where input point $p_{\delta}=(x_{\delta},y_{\delta},z_{\delta})$ is the relative point coordinate to the kernel center, and kernel-weight coordinate $p^{\prime}=(x^{\prime},y^{\prime},z^{\prime})$. We further note that Eq. (5) is a simplified format for normalized points. One property of trilinear interpolation is self-normalization, namely, all $8$ weights which an input point assigns to can sum up to $1$.

In Gaussian interpolation, we assign each input point $p_{\delta}$ to each kernel weight vector at $p^{\prime}$ with a weight factor calculated by the following Gaussian function,

where the hyperparameter $\sigma$ controls the decay rate. To save computation, if a 3D point is $3\sigma$ away from a weight vector, its assignment coefficient to the vector is directly set to and will not be calculated. It is worth noting that other functions, for example, linear basis functions, can also be adopted as the interpolation function.

Normalization terms. Given the fact that we take all neighboring points of a kernel weight vector into calculation, normalization is necessary to keep convolutions invariant to points density. There are two ways of normalization. We can aggregate and normalize the point features by

where $N$ is the number of neighboring points, $f_{i}$ is the $i$th point feature, and $t_{i}$ denotes its interpolation weight. Apart from normalizing according to the number of points, we can also normalize the sum of interpolation weights:

We can perform normalization either on each kernel weight vector or on the whole convolutional kernel. We argue that normalization on each kernel weight vector is more accurate, since input points are not uniformly distributed in the whole kernel.

The InterpConv Algorithm. An InterpConv operation takes point cloud coordinates and their features as input, and outputs new point coordinates and features. We note that output point coordinates can be set as the same as the input points, or downsampled from input point clouds. The center of convolutional kernels is placed at each output point coordinate, and kernel-weight coordinates are further determined by the relative coordinates of the weight vectors following Eq. (4). We calculate interpolation weights between kernel weight vectors and adjacent input points, and then aggregate feature by the weighted sum of all neighboring point features. The aggregated features are further normalized to keep it sparsity invariant. Finally dot production is applied between the normalized features and kernel weight vectors. A convolutional kernel sums all the results and $c^{\prime}$ kernels constitute a $1\times c^{\prime}$ new feature vector at the output coordinate. See the InterpConv algorithm for details.

3 Network Architectures

In this section, we introduce details for two deep architectures based on our InterpConv approaches. We explore embedding mutli-scale context features in the classification network and a deep encoder-decoder architecture in the segmentation network. See Figure 2 for details.

The classification network consists of a series of InterpConv blocks which is mainly composed of three InterpConv layers. In the InterpConv block, the first and last layer are of kernel size $1\times 1\times 1$ and the middle layer has a kernel size $3\times 3\times 3$. The first InterpConv layer reduces channel dimensions and the last InterpConv layer increases channel dimensions, leaving the middle InterpConv layer with relatively small input and output channels. One BatchNorm and ReLU layer also follow each InterpConv layer in the block.

Apart from this, we propose the PointInception module to encode multi-scale geometric features. Similar to the Inception module in 2D CNNs, our PointInception module also concatenates multi-branch features. However, we design each branch as one InterpConv block with a different kernel length $l$. The hyperparameter $l$ determines the distances between adjacent kernel weight vectors in the Euclidean space and controls the receptive field of the convolution. So the PointInception module is able to capture both fine-grained local structures and shape context information by combining multi-branch outputs. We further explore a deeper model by stacking two PointInception modules.

In the segmentation network, we share the similar spirit as U-Net and build a deep encoder-decoder architecture. We stack multiple $3\times 3\times 3$ InterpConv layers in the encoder, and in each layer output points are downsampled. In the first $3\times 3\times 3$ InterpConv layer, we set the kernel length $l$ as a small value in order to capture fine-grained geometric structures, which is important in semantic segmentation. We then gradually enlarge the kernel length $\mathit{l}$ in the following blocks to capture context information. For the upsampling layers in the decoder, we utilize feature propagation layers following . Skip connections are added between layers that have the same number of output points. The decoder outputs are then fed into an InterpConv layer with kernel size $1\times 1\times 1$ to obtain the final predictions.

Experiments

In this section, we evaluate the efficacy of Interpolated Convolutional Neural Networks on multiple tasks, including shape classification, object part segmentation and indoor scene semantic parsing. In all experiments, we implement the models using CUDA and PyTorch on NVIDIA TITAN X GPUs, and we use the Adam optimizer. We first demonstrate the performances of our approach on those tasks. Then we discuss key components of our method in ablation study.

Dataset. We evaluate the 3D shape classification performance of our network on the benchmark dataset ModelNet40 . ModelNet40 is composed of 12,311 CAD models which belong to 40 categories with 9,843 for training and 2,468 for testing. We use the point cloud conversion of ModelNet40, where 2,048 points are sampled from each CAD model. We further sample 1,024 points for training and testing following .

Implementation details. We adopt the classification network in Figure 2(a). We use Gaussian interpolation as our interpolation function and fix the Gaussian bandwidth $3\sigma$ to 0.1 in all InterpConv blocks. Point clouds are downsampled to half of the input number after each $3\times 3\times 3$ InterpConv layer. The input point clouds are randomly scaled by a factor ranging from 0.8 to 1.2, then jittered by a zero-mean Gaussian noise with 0.02 standard deviation. We trained the network for 480 epoches with initial learning rate 0.001 and decay rate 0.7 every 80 epoches with batch size $16$.

Results. We report the overall accuracy on this dataset. In Table 1, we compare our InterpCNN with other approaches. We demonstrate that the deep architecture based on InterpConvs performs much better than graph-based and voxel-based counterparts, with 0.8% improvement on the best graph-based network DGCNN . Our approach performs even better than Point2Seq and 3DCapsule in which many modules and model compacity are added on top of PointNet++ to gain a better performance.

2 Object Part Segmentation

Dataset. We evaluate our segmentation network on the part segmentation dataset ShapeNet Parts . ShapeNet Parts consists of 16,880 models from 16 shape categories, with 14,006 for training and 2,874 for testing. Each model is annotated with 2 to 6 parts and there are 50 different parts in total. Each point sampled from the models is annotated with a part label.

Implementation details. We use the segmentation network in Figure 2(b). During training we randomly sample 2,048 points from each object and use the original point clouds for testing. Different from the classification network, we utilize a trilinear interpolation function with a smaller kernel length $l$, which is shown to perform much better. The kernel length $l$ starts with 0.05 in the first InterpConv layer and doubles in the following layers. We use a minibatch of 32 in each GPU and 4 GPUs to train a model. We set the initial learning rate to be 0.005. Data augmentation is the same as classification.

Results. We report mean IOU over categories and instances in Table 2. It is worth noting that mean IOU over instances is more realistic. Our approach performs better than compared methods on mean IOU over instances.

3 Indoor Scene Segmentation

Dataset. S3DIS is an indoor sence semantic parsing dataset which contains 271 rooms in 6 areas. Each room is scanned by Matterport scanners and every point in the scan is annotated with one semantic label from 13 categories. We follow and split rooms into $1$m $\times$ $1$m blocks for training and testing.

Implementation details. Similar to the part segmentation task we use the same architecture in Figure 2(b). The difference is that we take 4,096 points from each $1$m $\times$ $1$m block as inputs during training. We construct a 9D vector (XYZ, RGB, and the normalized location) for each input. Other configurations are the same as that in the object part segmentation task.

Results. Following , we adpot 6-fold validations on 6 areas, and we report overall accuracy and mean IOU over categories in Table 3. Our approach significantly outperforms state-of-the-art methods on both accuracy and mean IOU.

4 Ablation Study

We perform ablation studies to investigate components of our InterpCNNs on ModelNet40 and ShapeNet Parts.

Effectiveness of kernel size $n$ and kernel length $l$. We explore different settings of hyperparameter $n$ and $l$ for the 1st and 2nd PointInception module in Table 4 and Table 5. We first try the setting of all InterpConvs with kernel size $1\times 1\times 1$ and only use a maxpooling layer to aggregate global features. We note that this architecture is similar to PointNet in which the network cannot capture local structures, and the result is much worse. This indicates the great power of InterpConvs with kernel size more than $1$. Simply replacing one $1\times 1\times 1$ InterpConv with $3\times 3\times 3$ for each InterpConv block obtains a performance gain of 3%. We also try InterpConvs with a larger kernel size $5\times 5\times 5$ but the performance does not improve. We demonstrate that utilizing $3\times 3\times 3$ InterpConvs is sufficiently effective and this also reduces model parameters compared with the $5\times 5\times 5$ counterparts. We also explore different kernel lengths and we show that this hyperparameter has a significant effect on the final performance. Either too small or too large the kernel length $l$ will harm the accuracy.

Effectiveness of interpolation functions. We try both Gaussian and trilinear interpolation functions in all tasks. In Table 6, the results show that Gaussian interpolation performs better in classification while trilinear interpolation is better in segmentaion. We argue that trilinear interpolation can capture fine-grained geometric structures better than Gaussian counterpart, which is more important for segmentation. Gaussian interpolation is able to obtain global shape information more effectively.

Effectiveness of normalization methods. We try normalization according to the number of neightboring points (Eq. (7)) and the sum of interoplation weights (Eq. (8)). The results in Table 7 indicate that both two methods are effective and show comparable performances. It is worth noting that in extreme cases where there are only a few close points but many far away points in the neighborhood of a kernel-weight coordinate, normalization on the sum of interpolation weights is more appropriate.

Model parameters analysis. We report the number of parameters in our classification network on ModelNet40. Results in Table 8 show that even with the comparable model parameters, PointNet++ still performs much worse than our approach. InterpCNNs also have fewer parameters than other 3D convolution methods.

Runtime analysis. We summarize average inference time based on the classification network with batch size $16$, $1024$ points on an NVIDIA TITAN X GPU, and compare it with pioneering work PointNet++ and DGCNN under the same settings. In Table 9, average inference time of our approach is slightly slower than PointNet++, but much faster than graph-based approach DGCNN.

Visualization. We visualize activations by different kernels in the first $3\times 3\times 3$ InterpConv layer in Figure 5 and some failure cases in Figure 6.

Conclusion

We propose a novel convolution InterpConv and Interpolated Convolutional Neural Networks (InterpCNNs) for 3D classification and segmentation. Experiments on ModelNet40, ShapeNet Parts and S3DIS show promising results compared with existing methods. For future work, we plan to explore new deep architectures based on learnable kernel-weight coordinates, and apply our approach on other point cloud processing tasks including 3D detection and instance segmentation.

Acknowledgements

This work is supported in part by SenseTime Group Limited, in part by the General Research Fund through the Research Grants Council of Hong Kong under Grants CUHK14202217, CUHK14203118, CUHK14205615, CUHK14207814, CUHK14213616, CUHK14208417, CUHK14239816, in part by CUHK Direct Grant.

References