Deep Learning for 3D Point Clouds: A Survey

Introduction

With the rapid development of 3D acquisition technologies, 3D sensors are becoming increasingly available and affordable, including various types of 3D scanners, LiDARs, and RGB-D cameras (such as Kinect, RealSense and Apple depth cameras) . 3D data acquired by these sensors can provide rich geometric, shape and scale information . Complemented with 2D images, 3D data provides an opportunity for a better understanding of the surrounding environment for machines. 3D data has numerous applications in different areas, including autonomous driving, robotics, remote sensing, and medical treatment .

3D data can usually be represented with different formats, including depth images, point clouds, meshes, and volumetric grids. As a commonly used format, point cloud representation preserves the original geometric information in 3D space without any discretization. Therefore, it is the preferred representation for many scene understanding related applications such as autonomous driving and robotics. Recently, deep learning techniques have dominated many research areas, such as computer vision, speech recognition, and natural language processing. However, deep learning on 3D point clouds still face several significant challenges , such as the small scale of datasets, the high dimensionality and the unstructured nature of 3D point clouds. On this basis, this paper focuses on the analysis of deep learning methods which have been used to process 3D point clouds.

Deep learning on point clouds has been attracting more and more attention, especially in the last five years. Several publicly available datasets are also released, such as ModelNet , ScanObjectNN , ShapeNet , PartNet , S3DIS , ScanNet , Semantic3D , ApolloCar3D , and the KITTI Vision Benchmark Suite . These datasets have further boosted the research of deep learning on 3D point clouds, with an increasingly number of methods being proposed to address various problems related to point cloud processing, including 3D shape classification, 3D object detection and tracking, 3D point cloud segmentation, 3D point cloud registration, 6-DOF pose estimation, and 3D reconstruction . Few surveys of deep learning on 3D data are also available, such as . However, our paper is the first to specifically focus on deep learning methods for point cloud understanding. A taxonomy of existing deep learning methods for 3D point clouds is shown in Fig. 1.

Compared with the existing literatures, the major contributions of this work can be summarized as follows:

To the best of our knowledge, this is the first survey paper to comprehensively cover deep learning methods for several important point cloud understanding tasks, including 3D shape classification, 3D object detection and tracking, and 3D point cloud segmentation.

As opposed to existing reviews , we specifically focus on deep learning methods for 3D point clouds rather than all types of 3D data.

This paper covers the most recent and advanced progresses of deep learning on point clouds. Therefore, it provides the readers with the state-of-the-art methods.

Comprehensive comparisons of existing methods on several publicly available datasets are provided (e.g., in Tables II, III, IV, V), with brief summaries and insightful discussions being presented.

The structure of this paper is as follows. Section 2 introduces the datasets and evaluation metrics for the respective tasks. Section 3 reviews the methods for 3D shape classification. Section 4 provides a survey of existing methods for 3D object detection and tracking. Section 5 presents a review of methods for point cloud segmentation, including semantic segmentation, instance segmentation, and part segmentation. Finally, Section 6 concludes the paper. We also provide a regularly updated project page on: https://github.com/QingyongHu/SoTA-Point-Cloud.

Background

A large number of datasets have been collected to evaluate the performance of deep learning algorithms for different 3D point clouds applications. Table I lists some typical datasets used for 3D shape classification, 3D object detection and tracking, and 3D point cloud segmentation. In particular, the attributes of these datasets are also summarized.

For 3D shape classification, there are two types of datasets: synthetic datasets and real-world datasets . Objects in the synthetic datasets are complete, without any occlusion and background. In contrast, objects in the real-world datasets are occluded at different levels and some objects are contaminated with background noise.

For 3D object detection and tracking, there are two types of datasets: indoor scenes and outdoor urban scenes . The point clouds in the indoor datasets are either converted from dense depth maps or sampled from 3D meshes. The outdoor urban datasets are designed for autonomous driving, where objects are spatially well separated and these point clouds are sparse.

For 3D point cloud segmentation, these datasets are acquired by different types of sensors, including Mobile Laser Scanners (MLS) , Aerial Laser Scanners (ALS) , static Terrestrial Laser Scanners (TLS) , RGB-D cameras and other 3D scanners . These datasets can be used to develop algorithms for various challenges including similar distractors, shape incompleteness, and class imbalance.

2 Evaluation Metrics

Different evaluation metrics have been proposed to test these methods for various point cloud understanding tasks. For 3D shape classification, Overall Accuracy (OA) and mean class accuracy (mAcc) are the most frequently used performance criteria. ‘OA’ represents the mean accuracy for all test instances and ‘mAcc’ represents the mean accuracy for all shape classes. For 3D object detection, Average Precision (AP) is the most frequently used criterion. It is calculated as the area under the precision-recall curve. Precision and Success are commonly used to evaluate the overall performance of a 3D single object tracker. Average Multi-Object Tracking Accuracy (AMOTA) and Average Multi-Object Tracking Precision (AMOTP) are the most frequently used criteria for the evaluation of 3D multi-object tracking. For 3D point cloud segmentation, OA, mean Intersection over Union (mIoU) and mean class Accuracy (mAcc) are the most frequently used criteria for performance evaluation. In particular, mean Average Precision (mAP) is also used in instance segmentation of 3D point clouds.

3D Shape Classification

Methods for this task usually learn the embedding of each point first and then extract a global shape embedding from the whole point cloud using an aggregation method. Classification is finally achieved by feeding the global embedding into several fully connected layers. According to the data type of input for neural networks, existing 3D shape classification methods can be divided into multi-view based, volumetric-based and point-based methods. Several milestone methods are illustrated in Fig. 2.

Multi-view based methods project an unstructured point cloud into 2D images, while volumetric-based methods convert a point cloud into a 3D volumetric representation. Then, well-established 2D or 3D convolutional networks are leveraged to achieve shape classification. In contrast, point-based methods directly work on raw point clouds without any voxelization or projection. Point-based methods do not introduce explicit information loss and become increasingly popular. Note that, this paper mainly focuses on point-based methods, but also includes few multi-view based and volumetric-based methods for completeness.

These methods first project a 3D shape into multiple views and extract view-wise features, and then fuse these features for accurate shape classification. How to aggregate multiple view-wise features into a discriminative global representation is a key challenge for these methods.

MVCNN is a pioneering work, which simply max-pools multi-view features into a global descriptor. However, max-pooling only retains the maximum elements from a specific view, resulting in information loss. MHBN integrates local convolutional features by harmonized bilinear pooling to produce a compact global descriptor. Yang et al. first leveraged a relation network to exploit the inter-relationships (e.g., region-region relationship and view-view relationship) over a group of views, and then aggregated these views to obtain a discriminative 3D object representation. In addition, several other methods have also been proposed to improve the recognition accuracy. Unlike previous methods, Wei et al. used a directed graph in View-GCN by considering multiple views as grpah nodes. The core layer composing of local graph convolution, non-local message passing and selective view-sampling is then applied to the constructed graph. The concatenation of max-pooled node features at all levels is finally used to form the global shape descriptor.

2 Volumetric-based Methods

These methods usually voxelize a point cloud into 3D grids, and then apply a 3D Convolution Neural Network (CNN) on the volumetric representation for shape classification.

Maturana et al. introduced a volumetric occupancy network called VoxNet to achieve robust 3D object recognition. Wu et al. proposed a convolutional deep belief-based 3D ShapeNets to learn the distribution of points from various 3D shapes (which are represented by a probability distribution of binary variables on voxel grids). Although encouraging performance has been achieved, these methods are unable to scale well to dense 3D data since the computation and memory footprint grow cubically with the resolution.

To this end, a hierarchical and compact structure (such as octree) is introduced to reduce the computational and memory costs of these methods. OctNet first hierarchically partitions a point cloud using a hybrid grid-octree structure, which represents the scene with several shallow octrees along a regular grid. The structure of octree is encoded efficiently using a bit string representation, and the feature vector of each voxel is indexed by simple arithmetic. Wang et al. proposed an Octree-based CNN for 3D shape classification. The average normal vectors of a 3D model sampled in the finest leaf octants are fed into the network, and 3D-CNN is applied on the octants occupied by the 3D shape surface. Compared to a baseline network based on dense input grids, OctNet requires much less memory and runtime for high-resolution point clouds. Le et al. proposed a hybrid network called PointGrid, which integrates the point and grid representation for efficient point cloud processing. A constant number of points is sampled within each embedding volumetric grid cell, which allows the network to extract geometric details by using 3D convolutions. Ben-Shabat et al. transformed the input point cloud into 3D grids which are further represented by 3D modified Fisher Vector (3DmFV) method, and then learned the global representation through a conventional CNN architecture.

3 Point-based Methods

According to the network architecture used for the feature learning of each point, methods in this category can be divided into pointwise MLP, convolution-based, graph-based, hierarchical data structure-based methods and other typical methods.

These methods model each point independently with several shared Multi-Layer Perceptrons (MLPs) and then aggregate a global feature using a symmetric aggregation function, as shown in Fig. 3.

Typical deep learning methods for 2D images cannot be directly applied to 3D point clouds due to their inherent data irregularities. As a pioneering work, PointNet directly takes point clouds as its input and achieves permutation invariance with a symmetric function. Specifically, PointNet learns pointwise features independently with several MLP layers and extracts global features with a max-pooling layer. Deep sets achieves permutation invariance by summing up all representations and applying nonlinear transformations. Since features are learned independently for each point in PointNet , the local structural information between points cannot be captured. Therefore, Qi et al. proposed a hierarchical network PointNet++ to capture fine geometric structures from the neighborhood of each point. As the core of PointNet++ hierarchy, its set abstraction level is composed of three layers: the sampling layer, the grouping layer and the PointNet based learning layer. By stacking several set abstraction levels, PointNet++ learns features from a local geometric structure and abstracts the local features layer by layer.

Because of its simplicity and strong representation ability, many networks have been developed based on PointNet . The architecture of Mo-Net is similar to PointNet but it takes a finite set of moments as its input. Point Attention Transformers (PATs) represents each point by its own absolute position and relative positions with respect to its neighbors and learns high dimensional features through MLPs. Then, Group Shuffle Attention (GSA) is used to capture relations between points, and a permutation invariant, differentiable and trainable end-to-end Gumbel Subset Sampling (GSS) layer is developed to learn hierarchical features. Based on PointNet++ , PointWeb utilizes the context of the local neighborhood to improve point features using Adaptive Feature Adjustment (AFA). Duan et al. proposed a Structural Relational Network (SRN) to learn structural relational features between different local structures using MLP. Lin et al. accelerated the inference process by constructing a lookup table for both input and function spaces learned by PointNet. The inference time on the ModelNet and ShapeNet datasets is sped up by 1.5 ms and 32 times over PointNet on a moderate machine. SRINet first projects a point cloud to obtain rotation invariant representations, and then utilizes PointNet-based backbone to extract a global feature and graph-based aggregation to extract local features. In PointASNL, Yan et al. utilized an Adaptive Sampling (AS) module to adaptively adjust the coordinates and features of points sampled by the Furthest Point Sampling (FPS) algorithm, and proposed a local-non-local (L-NL) module to capture the local and long range dependencies of these sampled points.

3.2 Convolution-based Methods

Compared with kernels defined on 2D grid structures (e.g., images), convolutional kernels for 3D point clouds are hard to design due to the irregularity of point clouds. According to the type of convolutional kernels, current 3D convolution methods can be divided into continuous and discrete convolution methods, as shown in Fig. 4.

3D Continuous Convolution Methods. These methods define convolutional kernels on a continuous space, where the weights for neighboring points are related to the spatial distribution with respect to the center point.

3D convolution can be interpreted as a weighted sum over a given subset. As the core layer of RS-CNN , RS-Conv takes a local subset of points around a certain point as its input, and the convolution is implemented using an MLP by learning the mapping from low-level relations (such as Euclidean distance and relative position) to high-level relations between points in the local subset. In , kernel elements are selected randomly in a unit sphere. An MLP-based continuous function is then used to establish relation between the locations of the kernel elements and the point cloud. In DensePoint , convolution is defined as a Single-Layer Perceptron (SLP) with a nonlinear activator. Features are learned by concatenating features from all previous layers to sufficiently exploit the contextual information. Thomas et al. proposed both rigid and deformable Kernel Point Convolution (KPConv) operators for 3D point clouds using a set of learnable kernel points. ConvPoint separates the convolution kernel into spatial and feature parts. The locations of the spatial part are randomly selected from a unit sphere and the weighting function is learned through a simple MLP.

Some methods also use existing algorithms to perform convolution. In PointConv , convolution is defined as a Monte Carlo estimation of the continuous 3D convolution with respect to an importance sampling. The convolutional kernels consist of a weighting function (which is learned with MLP layers) and a density function (which is learned by a kernelized density estimation and an MLP layer). To improve memory and computational efficiency, the 3D convolution is further reduced into two operations: matrix multiplication and 2D convolution. With the same parameter setting, its memory consumption can be reduced by about 64 times. In MCCNN , convolution is considered as a Monte Carlo estimation process relying on a sample’s density function (which is implemented with MLP). Poisson disk sampling is then used to construct a point cloud hierarchy. This convolution operator can be used to perform convolution between two or multiple sampling methods and can handle varying sampling densities. In SpiderCNN , SpiderConv is proposed to define convolution as the product of a step function and a Taylor expansion defined on the $k$ nearest neighbors. The step function captures the coarse geometry by encoding the local geodesic distance, and the Taylor expansion captures the intrinsic local geometric variations by interpolating arbitrary values at the vertices of a cube. Besides, a convolution network PCNN is also proposed for 3D point clouds based on the radial basis function.

Several methods have been proposed to address the rotation equivariant problem faced by 3D convolution networks. Esteves et al. proposed 3D Spherical CNN to learn rotation equivariant representation for 3D shapes, which takes multi-valued spherical functions as its input. Localized convolutional filters are obtained by parameterizing spectrum with anchor points in the spherical harmonic domain. Tensor field networks are proposed to define the point convolution operation as the product of a learnable radial function and spherical harmonics, which are locally equivariant to 3D rotations, translations, and permutations. The convolution in is defined based on the spherical cross-correlation and implemented using a generalized Fast Fourier Transformation (FFT) algorithm. Based on PCNN, SPHNet achieves rotation invariance by incorporating spherical harmonic kernels during convolution on volumetric functions.

To accelerate computing speed, Flex-Convolution defines weights of convolution kernel as standard scalar product over $k$ nearest neighbors, which can be accelerated using CUDA. Experimental results have demonstrated its competitive performance on a small dataset with fewer parameters and lower memory consumption.

3D Discrete Convolution Methods. These methods define convolutional kernels on regular grids, where the weights for neighboring points are related to the offsets with respect to the center point.

Hua et al. transformed non-uniform 3D point clouds into uniform grids and defined convolutional kernels on each grid. The proposed 3D kernel assigns the same weights to all points falling into the same grid. For a given point, the mean features of all the neighboring points that are located on the same grid are computed from the previous layer. Then, mean features of all grids are weighted and summed to produce the output of the current layer. Lei et al. defined a spherical convolutional kernel by partitioning a 3D spherical neighboring region into multiple volumetric bins and associating each bin with a learnable weighting matrix. The output of the spherical convolutional kernel for a point is determined by the non-linear activation of the mean of weighted activation values of its neighboring points. In GeoConv , the geometric relationship between a point and its neighboring points is explicitly modeled based on six bases. Edge features along each direction of the basis are weighted independently by a direction-associated learnable matrix. These direction-associated features are then aggregated according to the angles formed by the given point and its neighboring points. For a given point, its feature at the current layer is defined as the sum of features of the given point and its neighboring edge features at the previous layer.

PointCNN transforms the input points into a latent and potentially canonical order through a $\chi$-conv transformation (which is implemented through MLP) and then applies typical convolutional operator on the transformed features. By interpolating point features to neighboring discrete convolutional kernel-weight coordinates, Mao et al. proposed an interpolated convolution operator InterpConv to measure the geometric relations between input point clouds and kernel-weight coordinates. Zhang et al. proposed a RIConv operator to achieve rotation invariance, which takes low-level rotation invariant geometric features as input and then turns the convolution into 1D by a simple binning approach. A-CNN defines an annular convolution by looping the array of neighbors with respect to the size of kernel on each ring of the query point and learns the relationship between neighboring points in a local subset.

To reduce the computational and memory cost of 3D CNNs, Kumawat et al. proposed a Rectified Local Phase Volume (ReLPV) block to extract phase in a 3D local neighborhood based on 3D Short Term Fourier Transform (STFT), which significantly reduces the number of parameters. In SFCNN , a point cloud is projected onto regular icosahedral lattices with aligned spherical coordinates. Convolutions are then conducted upon the features concatenated from vertices of spherical lattices and their neighbors through convolution-maxpooling-convolution structures. SFCNN is resistant to rotations and perturbations.

3.3 Graph-based Methods

Graph-based networks consider each point in a point cloud as a vertex of a graph, and generate directed edges for the graph based on the neighbors of each point. Feature learning is then performed in spatial or spectral domains . A typical graph-based network is shown in Fig. 5.

Graph-based Methods in Spatial Domain. These methods define operations (e.g., convolution and pooling) in spatial domain. Specifically, convolution is usually implemented through MLP over spatial neighbors, and pooling is adopted to produce a new coarsened graph by aggregating information from each point’s neighbors. Features at each vertex are usually assigned with coordinates, laser intensities or colors, while features at each edge are usually assigned with geometric attributes between two connected points.

As a pioneering work, Simonovsky et al. considered each point as a vertex of the graph, and connected each vertex to all its neighbors by a directed edge. Then, Edge-Conditioned Convolution (ECC) is proposed using a filter-generating network (e.g., MLP). Max pooling is adopted to aggregate neighborhood information and graph coarsening is implemented based on VoxelGrid . In DGCNN , a graph is constructed in the feature space and dynamically updated after each layer of the network. As the core layer of EdgeConv, an MLP is used as the feature learning function for each edge, and channel-wise symmetric aggregation is applied onto the edge features associated with the neighbors of each point. Further, LDGCNN removes the transformation network and links the hierarchical features from different layers in DGCNN to improve its performance and reduce the model size. An end-to-end unsupervised deep AutoEncoder network (namely, FoldingNet ) is also proposed to use the concatenation of a vectorized local covariance matrix and point coordinates as its input. Inspired by Inception and DGCNN , Hassani and Haley proposed an unsupervised multi-task autoencoder to learn point and shape features. The encoder is constructed based on mutli-scale graphs. The decoder is constructed using three unsupervised tasks including clustering, self-supervised classification and reconstruction, which are trained jointly with a mutli-task loss. Liu et al. proposed a Dynamic Points Agglomeration Module (DPAM) based on graph convolution to simplify the process of points agglomeration (sampling, grouping and pooling) into a simple step, which is implemented through multiplication of the agglomeration matrix and points feature matrix. Based on the PointNet architecture, a hierarchical learning architecture is constructed by stacking multiple DPAMs. Compared with the hierarchy strategy of PointNet++ , DPAM dynamically exploits the relation of points and agglomerates points in a semantic space.

To exploit the local geometric structures, KCNet learns features based on kernel correlation. Specifically, a set of learnable points characterizing geometric types of local structures are defined as kernels. Then, affinity between the kernel and the neighborhood of a given point is calculated. In G3D , convolution is defined as a variant of polynomial of adjacency matrix, and pooling is defined as multiplying the Laplacian matrix and the vertex matrix by a coarsening matrix. ClusterNet utilizes a rigorously rotation-invariant module to extract rotation-invariant features from $k$ nearest neighbors for each point, and constructs hierarchical structures of a point cloud based on the unsupervised agglomerative hierarchical clustering method with ward-linkage criteria . The features in each sub-cluster are first learned through an EdgeConv block and then aggregated through max pooling.

To address the time-consuming problem of current data structuring methods (such as FPS and neighbor points querying), Xu et al. proposed to blend the advantages of volumetric based and point based methods to improve the computational efficiency. Experiments on the ModelNet classification task demonstrate that the computational efficiency of the proposed Grid-GCN network is 5$\times$ faster than other models in average.

Graph-based Methods in Spectral Domain. These methods define convolutions as spectral filtering, which is implemented as the multiplication of signals on graph with eigenvectors of the graph Laplacian matrix .

RGCNN constructs a graph by connecting each point with all other points in the point cloud and updates the graph Laplacian matrix in each layer. To make features of adjacent vertices more similar, a graph-signal smoothness prior is added into the loss function. To address the challenges caused by diverse graph topology of data, the SGC-LL layer in AGCN utilizes a learnable distance metric to parameterize the similarity between two vertices on the graph. The adjacency matrix obtained from graph is normalized using Gaussian kernels and learned distances. HGNN builds a hyperedge convolutional layer by applying spectral convolution on a hypergraph.

Aforementioned methods operate on full graphs. To exploit local structural information, Wang et al. proposed an end-to-end spectral convolution network LocalSpecGCN to work on a local graph (which is constructed from the $k$ nearest neighbors). This method does not require any offline computation of the graph Laplacian matrix and graph coarsening hierarchy. In PointGCN , a graph is constructed based on $k$ nearest neighbors from a point cloud and each edge is weighted using a Gaussian kernel. Convolutional filters are defined as Chebyshev polynomials in graph spectral domain. Global pooling and multi-resolution pooling are used to capture global and local features of the point cloud. Pan et al. proposed 3DTI-Net by applying convolution on the $k$ nearest neighboring graphs in spectral domain. The invariance to geometry transformation is achieved by learning from relative Euclidean and direction distances.

3.4 Hierarchical Data Structure-based Methods

These networks are constructed based on different hierarchical data structures (e.g., octree and kd-tree). In these methods, point features are learned hierarchically from leaf nodes to the root node along a tree.

Lei et al. proposed an octree guided CNN using spherical convolutional kernels (as described in Section 4). Each layer of the network corresponds to one layer of the octree and a spherical convolutional kernel is applied at each layer. The values of neurons in the current layer are determined as the mean values of all relevant children nodes in the previous layer. Unlike OctNet which is based on octree, Kd-Net is built using multiple K-d trees with different splitting directions at each iteration. Following a bottom-up approach, the representation of a non-leaf node is computed from representations of its children using MLP. The feature of the root node (which describes the whole point cloud) is finally fed to fully connected layers to predict classification scores. Note that, Kd-Net shares parameters at each level according to the splitting type of nodes. 3DContextNet uses a standard balanced K-d tree to achieve feature learning and aggregation. At each level, point features are first learned through MLP based on local cues (which models inter-dependencies between points in a local region) and global contextual cues (which models the relationship for one position with respect to all other positions). Then, the feature of a non-leaf node is computed from its child nodes using MLP and aggregated by max pooling. For classification, the above process is repeated until the root node is attained.

The hierarchy of SO-Net network is constructed by performing point-to-node $k$ nearest neighbor search . Specifically, a modified permutation invariant Self-Organizing Map (SOM) is used to model the spatial distribution of a point cloud. Individual point features are learned from normalized point-to-node coordinates through a series of fully connected layers. The feature of each node in SOM is extracted from point features associated with this node using channel-wise max pooling. The final feature is then learned from node features using an approach similar to PointNet . Compared to PointNet++ , the hierarchy of SOM is more efficient and the spatial distribution of the point cloud is fully explored.

3.5 Other Methods

In addition, many other schemes have also been proposed. RBFNet explicitly models the spatial distribution of points by aggregating features from sparsely distributed Radial Basis Function (RBF) kernels with learnable kernel positions and sizes. 3DPointCapsNet learns point independent features with pointwise MLP and convolutional layers, and extracts global latent representation with multiple max-pooling layers. Based on unsupervised dynamic routing, powerful representative latent capsules are then learned. Qin et al. proposed an end-to-end unsupervised domain adaptation network PointDAN for 3D point cloud representation. To capture semantic properties of a point cloud, a self-supervised method is proposed to reconstruct the point cloud, whose parts have been randomly rearranged . Li et al. proposed an auto-augmentation framework, PointAugment, to automatically optimize and augment point cloud samples for network training. Specifically, shape-wise transformation and point-wise displacement for each input sample are automatically learned, and the network is trained by alternatively optimizing and updating the learnable parameters of its augmentor and classifier. Inspired by shape context , Xie et al. proposed a ShapeContextNet architecture by combining affinity point selection and compact feature aggregation into a soft alignment operation using dot-product self-attention . To handle noise and occlusion in 3D point clouds, Bobkov et al. fed handcrafted point pair function based 4D rotation invariant descriptors into a 4D convolutional neural network. Prokudin et al. first randomly sampled a basis point set with a uniform distribution from a unit ball, and then encoded a point cloud as minimal distances to the basis point set. Consequently, the point cloud is converted to a vector with a relatively small fixed length. The encoded representation can then be processed with existing machine learning methods.

RCNet utilizes standard RNN and 2D CNN to construct a permutation-invariant network for 3D point cloud processing. The point cloud is first partitioned into parallel beams and sorted along a specific dimension, and each beam is then fed into a shared RNN. The learned features are further fed into an efficient 2D CNN for hierarchical feature aggregation. To enhance its description ability, RCNet-E is proposed to ensemble multiple RCNets along different partition and sorting directions. Point2Sequences is another RNN-based model that captures correlations between different areas in local regions of point clouds. It considers features learned from a local region at multiple scales as sequences and feeds these sequences from all local regions into an RNN-based encoder-decoder structure to aggregate local region features.

Several methods also learn from both 3D point clouds and 2D images. In PVNet , high-level global features extracted from multi-view images are projected into the subspace of point clouds through an embedding network, and fused with point cloud features through a soft attention mask. Finally, a residual connection is employed for fused features and multi-view features to perform shape recognition. Later, PVRNet is further proposed to exploit the relation between a 3D point cloud and its multiple views by a relation score module. Based on the relation scores, the original 2D global view features are enhanced for point-single-view fusion and point-multi-view fusion.

4 Summary

The ModelNet10/40 datasets are the most frequently used datasets for 3D shape classification. Table II shows the results achieved by different point-based networks. Several observations can be drawn:

Pointwise MLP networks are usually served as the basic building block for other types of networks to learn pointwise features.

As a standard deep learning architecture, convolution-based networks can achieve superior performance on irregular 3D point clouds. More attention should be paid to both discrete and continuous convolution networks for irregular data.

Due to its inherent strong capability to handle irregular data, graph-based networks have attracted increasingly more attention in recent years. However, it is still challenging to extend graph-based networks in the spectral domain to various graph structures.

3D Object Detection and Tracking

In this section, we will review existing methods for 3D object detection, 3D object tracking and 3D scene flow estimation.

A typical 3D object detector takes the point cloud of a scene as its input and produces an oriented 3D bounding box around each detected object, as shown in Fig. 6. Similar to object detection in images , 3D object detection methods can be divided into two categories: region proposal-based and single shot methods. Several milestone methods are presented in Fig. 7.

These methods first propose several possible regions (also called proposals) containing objects, and then extract region-wise features to determine the category label of each proposal. According to their object proposal generation approach, these methods can further be divided into three categories: multi-view based, segmentation-based and frustum-based methods.

Multi-view based Methods. These methods fuse proposal-wise features from different view maps (e.g., LiDAR front view, Bird’s Eye View (BEV), and image) to obtain 3D rotated boxes, as shown in Fig. 8(a). The computational cost of these methods is usually high.

Chen et al. generated a group of highly accurate 3D candidate boxes from the BEV map and projected them to the feature maps of multiple views (e.g., LiDAR front view image, RGB image). They then combined these region-wise features from different views to predict oriented 3D bounding boxes, as shown in Fig. 8(a). Although this method achieves a recall of 99.1% at an Intersection over Union (IoU) of 0.25 with only 300 proposals, its speed is too slow for practical applications. Subsequently, several approaches have been developed to improve multi-view 3D object detection methods from two aspects.

First, several methods have been proposed to efficiently fuse the information of different modalities. To generate 3D proposals with a high recall for small objects, Ku et al. proposed a multi-modal fusion-based region proposal network. They first extracted equal-sized features from both BEV and image views using cropping and resizing operations, and then fused these features using element-wise mean pooling. Liang et al. exploited continuous convolutions to enable effective fusion of image and 3D LiDAR feature maps at different resolutions. Specifically, they extracted nearest corresponding image features for each point in the BEV space and then used bilinear interpolation to obtain a dense BEV feature map by projecting image features into the BEV plane. Experimental results show that dense BEV feature maps are more suitable for 3D object detection than discrete image feature maps and sparse LiDAR feature maps. Liang et al. presented a multi-task multi-sensor 3D object detection network for end-to-end training. Specifically, multiple tasks (e.g., 2D object detection, ground estimation and depth completion) are exploited to help the network learn better feature representations. The learned cross-modality representation is further exploited to produce highly accurate object detection results. Experimental results show that this method achieves a significant improvement on 2D, 3D and BEV detection tasks, and outperforms previous state-of-the-art methods on the TOR4D benchmark .

Second, different methods have been investigated to extract robust representations of the input data. Lu et al. explored multi-scale contextual information by introducing a Spatial Channel Attention (SCA) module, which captures the global and multi-scale context of a scene and highlights useful features. They also proposed an Extension Spatial Unsample (ESU) module to obtain high-level features with rich spatial information by combining multi-scale low-level features, thus generating reliable 3D object proposals. Although better detection performance can be achieved, the aforementioned multi-view methods take a long runtime since they perform feature pooling for each proposal. Subsequently, Zeng et al. used a pre-RoI pooling convolution to improve the efficiency of . Specifically, they moved the majority of convolution operations to be ahead of the RoI pooling module. Therefore, RoI convolutions are performed once for all object proposals. Experimental results show that this method can run at a speed of 11.1 fps, which is 5 times faster than MV3D .

Segmentation-based Methods. These methods first leverage existing semantic segmentation techniques to remove most background points, and then generate a large amount of high-quality proposals on foreground points to save computation, as shown in Fig. 8(b). Compared to multi-view methods , these methods achieve higher object recall rates and are more suitable for complicated scenes with highly occluded and crowded objects.

Yang et al. used a 2D segmentation network to predict foreground pixels and projected them into point clouds to remove most background points. They then generated proposals on the predicted foreground points and designed a new criterion named PointsIoU to reduce the redundancy and ambiguity of proposals. Following , Shi et al. proposed a PointRCNN framework. Specifically, they directly segmented 3D point clouds to obtain foreground points and then fused semantic features and local spatial features to produce high-quality 3D boxes. Following the Region Proposal Network (RPN) stage of , Jesus et al. proposed a pioneering work to leverage Graph Convolution Network (GCN) for 3D object detection. Specifically, two modules are introduced to refine object proposals using graph convolution. The first module R-GCN utilizes all points contained in a proposal to achieve per-proposal feature aggregation. The second module C-GCN fuses per-frame information from all proposals to regress accurate object boxes by exploiting contexts. Sourabh et al. projected a point cloud into the output of the image-based segmentation network and appended the semantic prediction scores to the points. The painted points are fed into existing detectors to achieve significant performance improvement. Yang et al. associated each point with a spherical anchor. The semantic score of each point is then used to remove redundant anchors. Consequently, this method achieves a higher recall with lower computational cost as compared to previous methods . In addition, a PointsPool layer is proposed to learn compact features for interior points in proposals and a parallel IoU branch is introduced to improve localization accuracy and detection performance.

Frustum-based Methods. These methods first leverage existing 2D object detectors to generate 2D candidate regions of objects and then extract a 3D frustum proposal for each 2D candidate region, as shown in Fig. 8(c). Although these methods can efficiently propose possible locations of 3D objects, the step-by-step pipeline makes their performance limited by 2D image detectors.

F-PointNets is a pioneering work in this direction. It generates a frustum proposal for each 2D region and applies PointNet (or PointNet++ ) to learn point cloud features of each 3D frustum for amodal 3D box estimation. In a follow-up work, Zhao et al. proposed a Point-SENet module to predict a set of scaling factors, which were further used to adaptively highlight useful features and suppress informative-less features. They also integrated the PointSIFT module into the network to capture orientation information of point clouds, which achieved strong robustness to shape scaling. This method achieves significant improvement on both indoor and outdoor datasets as compared to F-PointNets .

Xu et al. leveraged both 2D image region and its corresponding frustum points to accurately regress 3D boxes. To fuse image features and global features of point clouds, they presented a global fusion network for direct regression of box corner locations. They also proposed a dense fusion network for the prediction of point-wise offsets to each corner. Shin et al. first estimated 2D bounding boxes and 3D poses of objects from a 2D image, and then extracted multiple geometrically feasible object candidates. These 3D candidates are fed into a box regression network to predict accurate 3D object boxes. Wang et al. generated a sequence of frustums along the frustum axis for each 2D region and applied PointNet to extract features for each frustum. The frustum-level features are reformed to generate a 2D feature map, which is then fed into a fully convolutional network for 3D box estimation. This method achieves the state-of-the-art performance among 2D image-based methods and was ranked in the top position of the official KITTI leaderboard. Johannes et al. first obtained a preliminary detection results on the BEV map, and then extracted small point subsets (also called patches) based on the BEV predictions. A local refinement network is applied to learn the local features of patches to predict highly accurate 3D bounding boxes.

Other Methods. Motivated by the success of axis-aligned IoU in object detection in images, Zhou et al. integrated the IoU of two 3D rotated bounding boxes into several state-of-the-art detectors to achieve consistent performance improvement. Chen et al. proposed a two-stage network architecture to use both point cloud and voxel representations. First, point clouds are voxelized and fed to a 3D backbone network to produce initial detection results. Second, the interior point features of initial predictions are further exploited for box refinements. Although this design is conceptually simple, it achieves comparable performance to while maintaining a speed of 16.7 fps. Shi et al. proposed PointVoxel-RCNN (PV-RCNN) to leverage both 3D convolutional network and PointNet-based set abstraction for the learning of point cloud features. Specifically, the input point clouds are first voxelized and then fed into a 3D sparse convolutional network to generate high-quality proposals. The learned voxel-wise features are then encoded into a small set of key points via a voxel set abstraction module. In addition, they also proposed a keypoint-to-grid ROI abstraction module to capture rich context information for box refinement. Experimental results show that this method outperforms previous methods by a remarkable margin and is ranked firstThe ranking refers to the time of the submission: 12th June, 2020 on the $Car$ class of the KITTI 3D detection benchmark.

Inspired by Hough voting-based 2D object detectors, Qi et al. proposed VoteNet to directly vote for virtual center points of objects from point clouds and to generate a group of high-quality 3D object proposals by aggregating vote features. VoteNet significantly outperforms previous approaches using only geometric information, and achieves the state-of-the-art performance on two large indoor benchmarks (i.e., ScanNet and SUN RGB-D ). However, the prediction of virtual center point is unstable for a partially occluded object. Further, Feng et al. added an auxiliary branch of direction vectors to improve the prediction accuracy of virtual center points and 3D candidate boxes. In addition, a 3D object-object relationship graph between proposals is built to emphasize useful features for accurate object detection. Qi et al. proposed an ImVoteNet detector by fusing 2D object detection cues (e.g., geometric and semantic/texture cues) into a 3D voting pipeline. Inspired by the observation that the ground truth boxes of 3D objects provide accurate locations of intra-object parts, Shi et al. proposed the Part-$A^{2}$ Net, which is composed of a part-aware stage and a part-aggregation stage. The part-aware stage applies a UNet-like network with sparse convolution and sparse deconvolution to learn point-wise features for the prediction and coarse generation of intra-object part locations. The part-aggregation stage adopts RoI-aware pooling to aggregate predicted part locations for box refinement.

1.2 Single Shot Methods

These methods directly predict class probabilities and regress 3D bounding boxes of objects using a single-stage network. They do not need region proposal generation and post-processing. As a result, they can run at a high speed. According to the type of input data, single shot methods can be divided into three categories: BEV-based, discretization-based and point-based methods.

BEV-based Methods. These methods mainly take BEV representation as their input. Yang et al. discretized the point cloud of a scene with equally spaced cells and encoded the reflectance in a similar way, resulting in a regular representation. A Fully Convolution Network (FCN) network is then applied to estimate the locations and heading angles of objects. This method outperforms most single shot methods (including VeloFCN , 3D-FCN and Vote3Deep ) while running at 28.6 fps. Later, Yang et al. exploited the geometric and semantic prior information provided by High-Definition (HD) maps to improve the robustness and detection performance of . Specifically, they obtained the coordinates of ground points from the HD map and then used the distance relative to the ground for BEV representation to remedy the translation variance caused by the slope of the road. In addition, they concatenated a binary road mask with the BEV representation along the channel dimension to focus on moving objects. Since HD maps are not available everywhere, they also proposed an online map prediction module to estimate the map priors from single LiDAR point cloud. This map-aware method significantly outperforms its baseline on the TOR4D and KITTI datasets. However, its generalization performance to point clouds with different densities is poor. To solve this problem, Beltrán et al. proposed a normalization map to consider the differences among different LiDAR sensors. The normalization map is a 2D grid with the same resolution as the BEV map, and it encodes the maximum number of points contained in each cell. It is shown that this normalization map significantly improves the generalization ability of BEV-based detectors.

Discretization-based Methods. These methods convert a point cloud into a regular discrete representation, and then apply CNN to predict both categories and 3D boxes of objects.

Li et al. proposed the first method to use a FCN for 3D object detection. They converted a point cloud into a 2D point map and used a 2D FCN to predict the bounding boxes and confidences of objects. Later, they discretized the point cloud into a 4D tensor with dimensions of length, width, height and channels, and extended the 2D FCN-based detection technologies to 3D domain for 3D object detection. Compared to , 3D FCN-based method obtains a gain of over 20% in accuracy, but inevitably costs more computing resources due to 3D convolutions and the sparsity of the data. To address the sparsity problem of voxels, Engelcke et al. leveraged a feature-centric voting scheme to generate a set of votes for each non-empty voxel and to obtain the convolutional results by accumulating the votes. Its computational complexity is proportional to the number of occupied voxels. Li et al. constructed a 3D backbone network by stacking multiple sparse 3D CNNs. This method is designed to save memory and accelerate computation by fully using the sparsity of voxels. This 3D backbone network extracts rich 3D features for object detection without introducing heavy computational burden.

Zhou et al. presented a voxel-based end-to-end trainable framework VoxelNet. They partitioned a point cloud into equally spaced voxels and encoded the features within each voxel into a 4D tensor. A region proposal network is then connected to produce detection results. Although its performance is strong, this method is very slow due to the sparsity of voxels and 3D convolutions. Later, Yan et al. used the sparse convolutional network to improve the inference efficiency of . They also proposed a sine-error angle loss to solve the ambiguity between orientations of 0 and $\pi$. Sindagi et al. extended VoxelNet by fusing image and point cloud features at early stages. Specifically, they projected non-empty voxels generated by into the image and used a pre-trained network to extract image features for each projected voxel. These image features are then concatenated with voxel features to produce accurate 3D boxes. Compared to , this method can effectively exploit multi-modal information to reduce false positives and negatives. Lang et al. proposed a 3D object detector named PointPillars. This method leverages PointNet to learn the feature of point clouds organized in vertical columns (Pillars) and encodes the learned features as a pesudo image. A 2D object detection pipeline is then applied to predict 3D bounding boxes. PointPillars outperforms most fusion approaches (including MV3D , RoarNet and AVOD ) in terms of Average Precision (AP). Moreover, PointPillars can run at a speed of 62 fps on both the 3D and BEV KITTI benchmarks, making it highly suitable for practical applications.

Inspired by the observation that partial spatial information of a point cloud is inevitably lost in progressively downscaled feature maps of existing single shot detectors, He et al. proposed a SA-SSD detector to leverage the fine-grained structure information to improve localization accuracy. Specifically, they first converted a point cloud to a tensor and fed it into a backbone network to extract multi-stage features. In addition, an auxiliary network with point-level supervision is employed to guide the features to learn the structure of point clouds. Experimental results show that SA-SSD ranks the firstThe ranking refers to the time of the submission: 12th June, 2020 on the $Car$ class of the KITTI BEV detection benchmark.

Point-based Methods. These methods directly take raw point clouds as their inputs. 3DSSD is a pioneering work in this direction. It introduces a fusion sampling strategy for Distance-FPS (D-FPS) and Feature-FPS (F-FPS) to remove time-consuming Feature Propagation (FP) layers and the refinement module in . Then, a Candidate Generation (CG) layer is used to fully exploit representative points, which are further fed into an anchor-free regression head with a 3D centerness label to predict 3D object boxes. Experimental results show that 3DSSD outperforms the two-stage point-based method PointRCNN while maintaining a speed of 25 fps.

Other Methods. Meyer et al. proposed an efficient 3D object detector called LaserNet. This method predicts a probability distribution over bounding boxes for each point and then combines these per-point distributions to generate final 3D object boxes. Further, the dense Range View (RV) representation of point cloud is used as input and a fast mean-shift algorithm is proposed to reduce the noise produced by per-point prediction. LaserNet achieves the state-of-the-art performance at the range of 0 to 50 meters, and its runtime is significantly lower than existing methods. Meyer et al. then extended LaserNet to exploit the dense texture provided by RGB images (e.g., 50 to 70 meters). Specifically, they associated LiDAR points with image pixels by projecting 3D point clouds onto 2D images and exploited this association to fuse RGB information into 3D points. They also considered 3D semantic segmentation as an auxiliary task to learn better representations. This method achieves a significant improvement in both long-range (e.g., 50 to 70 meters) object detection and semantic segmentation while maintaining high efficiency of LaserNet. Inspired by the observation that points on an isolated object part can provide abundant information about position and orientation of the object, Chen et al. proposed a novel $Hotspot$ representation and the first hotspot-based anchor-free detector. Specifically, raw point clouds are first voxelized and then fed into a backbone network to produce 3D feature maps. These feature maps are used to classify hotspots and predict 3D bounding boxes simultaneously. Note that, hotspots are assigned at the last convolutional layer of the backbone network. Experimental results show that this method achieves comparable performance and is robust to sparse point clouds. Shi et el. proposed a graph neural network Point-GNN to detect 3D objects from lidar point clouds. They first encoded an input point cloud as a graph of near neighbors with a fixed radius and then fed the graph into Point-GNN to predict both the categories and boxes of objects.

2 3D Object Tracking

Given the locations of an object in the first frame, the task of object tracking is to estimate its state in subsequent frames . Since 3D object tracking can use the rich geometric information in point clouds, it is expected to overcome several drawbacks faced by image-based tracking, including occlusion, illumination and scale variation.

3 3D Scene Flow Estimation

Given two point clouds $\mathcal{X}$ and $\mathcal{Y}$, 3D scene flow $D=\{d_{i}\}^{N}$ describes the movement of each point $x_{i}$ in $\mathcal{X}$ to its corresponding position $x_{i}^{\prime}$ in $\mathcal{Y}$, such that $x_{i}^{\prime}=x_{i}+d_{i}$. Figure 9 shows a 3D scene flow between two KITTI point clouds. Analogous to optical flow estimation in 2D vision, several methods have started to learn useful information (e.g. 3D scene flow, spatial-temporary information) from a sequence of point clouds.

Liu et al. proposed FlowNet3D to directly learn scene flows from a pair of consecutive point clouds. FlowNet3D learns both point-level features and motion features through a flow embedding layer. However, there are two problems with FlowNet3D. First, some predicted motion vectors differ significantly from the ground truth in their directions. Second, it is difficult to apply FlowNet to non-static scenes, especially for the scenes which are dominated by deformable objects. To solve this problem, Wang et al. introduced a cosine distance loss to minimize the angle between the predictions and the ground truth. In addition, they also proposed a point-to-plane distance loss to improve the accuracy for both rigid and dynamic scenes. Experimental results show that these two loss terms improve the accuracy of FlowNet3D from 57.85% to 63.43%, and speed up and stabilize the training process. Gu et al. proposed a Hierarchical Permutohedral Lattice FlowNet (HPLFlowNet) to directly estimate scene flow from large-scale point clouds. Several bilateral convolution layers are proposed to restore structural information from raw point clouds, while reducing the computational cost.

To effectively process sequential point clouds, Fan and Yang proposed PointRNN, PointGRU and PointLSTM networks and a sequence-to-sequence model to track moving points. PointRNN, PointGRU, and PointLSTM are able to capture the spatial-temporary information and model dynamic point clouds. Similarly, Liu et al. proposed MeteorNet to directly learn a representation from dynamic point clouds. This method learns to aggregate information from spatiotemporal neighboring points. Direct grouping and chained-flow grouping are further introduced to determine the temporal neighbors. However, the performance of the aforementioned methods is limited by the scale of datasets. Mittal et al. proposed two self-supervised losses to train their network on large unlabeled datasets. Their main idea is that a robust scene flow estimation method should be effective in both forward and backward predictions. Due to the unavailability of scene flow annotation, the nearest neighbor of the predicted transformed point is considered as pesudo ground truth. However, the true ground truth may not be the same as the nearest point. To avoid this problem, they computed the scene flow in the reverse direction and proposed a cycle consistency loss to translate the point to the original position. Experimental results show that this self-supervised method exceeds the state-of-the-art performance of supervised learning-based methods.

4 Summary

The KITTI benchmark is one of the most influential datasets in autonomous driving and has been commonly used in both academia and industry. Tables III and IV present the results achieved by different detectors on the KITTI test 3D benchmarks. The following observations can be made:

Region proposal-based methods are the most frequently investigated methods among these two categories, and outperform single shot methods by a large margin on both KITTI test 3D and BEV benchmarks.

There are two limitations for existing 3D object detectors. First, the long-range detection capability of existing methods is relatively poor. Second, how to fully exploit the texture information in images is still an open problem.

Multi-task learning is a future direction in 3D object detection. E.g., MMF learns a cross-modality representation to achieve state-of-the-art detection performance by incorporating multiple tasks.

3D object tracking and scene flow estimation are emerging research topics, and have gradually attracted increasing attention since 2019.

3D Point Cloud Segmentation

3D point cloud segmentation requires the understanding of both the global geometric structure and the fine-grained details of each point. According to the segmentation granularity, 3D point cloud segmentation methods can be classified into three categories: semantic segmentation (scene level), instance segmentation (object level) and part segmentation (part level).

Given a point cloud, the goal of semantic segmentation is to separate it into several subsets according to the semantic meanings of points. Similar to the taxonomy for 3D shape classification (Section 3), there are four paradigms for semantic segmentation: projection-based, discretization-based, point-based, and hybrid methods.

The first step of both the projection and discretization-based methods is to transform a point cloud to an intermediate regular representation, such as multi-view , spherical , volumetric , permutohedral lattice , and hybrid representations , as shown in Fig. 11. The intermediate segmentation results are then projected back to the raw point cloud. In contrast, point-based methods directly work on irregular point clouds. Several representative methods are shown in Fig. 10.

These methods usually project a 3D point cloud into 2D images, including multi-view and spherical images.

Multi-view Representation. Lawin et al. first projected a 3D point cloud onto 2D planes from multiple virtual camera views. Then, a multi-stream FCN is used to predict pixel-wise scores on synthetic images. The final semantic label of each point is obtained by fusing the re-projected scores over different views. Similarly, Boulch et al. first generated several RGB and depth snapshots of a point cloud using multiple camera positions. They then performed pixel-wise labeling on these snapshots using 2D segmentation networks. The scores predicted from RGB and depth images are further fused using residual correction . Based on the assumption that point clouds are sampled from locally Euclidean surfaces, Tatarchenko et al. introduced tangent convolutions for dense point cloud segmentation. This method first projects the local surface geometry around each point to a virtual tangent plane. Tangent convolutions are then directly operated on the surface geometry. This method shows great scalability and is able to process large-scale point clouds with millions of points. Overall, the performance of multi-view segmentation methods is sensitive to viewpoint selection and occlusions. Besides, these methods have not fully exploited the underlying geometric and structural information, as the projection step inevitably introduces information loss.

Spherical Representation. To achieve fast and accurate segmentation of 3D point clouds, Wu et al. proposed an end-to-end network based on SqueezeNet and Conditional Random Field (CRF). To further improve segmentation accuracy, SqueezeSegV2 is introduced to address domain shift by utilizing an unsupervised domain adaptation pipeline. Milioto et al. proposed RangeNet++ for real-time semantic segmentation of LiDAR point clouds. The semantic labels of 2D range images are first transferred to 3D point clouds, an efficient GPU-enabled KNN-based post-processing step is further used to alleviate the problem of discretization errors and blurry inference outputs. Compared to single view projection, spherical projection retains more information and is suitable for the labeling of LiDAR point clouds. However, this intermediate representation inevitably brings several problems such as discretization errors and occlusions.

1.2 Discretization-based Methods

These methods usually convert a point cloud into a dense/sparse discrete representation, such as volumetric and sparse permutohedral lattices.

Dense Discretization Representation. Early methods usually voxelized the point clouds as dense grids and then leverage the standard 3D convolutions. Huang et al. first divided a point cloud into a set of occupancy voxels, then fed these intermediate data to a fully-3D CNN for voxel-wise segmentation. Finally, all points within a voxel are assigned the same semantic label as the voxel. The performance of this method is severely limited by the granularity of the voxels and the boundary artifacts caused by the point cloud partition. Further, Tchapmi et al. proposed SEGCloud to achieve fine-grained and global consistent semantic segmentation. This method introduces a deterministic trilinear interpolation to map the coarse voxel predictions generated by 3D-FCNN back to the point cloud, and then uses Fully Connected CRF (FC-CRF) to enforce spatial consistency of these inferred per-point labels. Meng et al. introduced a kernel-based interpolated variational autoencoder architecture to encode the local geometrical structures within each voxel. Instead of a binary occupancy representation, RBFs are employed for each voxel to obtain a continuous representation and capture the distribution of points in each voxel. VAE is further used to map the point distribution within each voxel to a compact latent space. Then, both symmetry groups and an equivalence CNN are used to achieve robust feature learning.

Thanks to the good scalability of 3D CNN, volumetric-based networks are free to be trained and tested on point clouds with different spatial sizes. In Fully-Convolutional Point Network (FCPN) , different levels of geometric relations are first hierarchically abstracted from point clouds, 3D convolutions and weighted average pooling are then used to extract features and incorporate long-range dependencies. This method can process large-scale point clouds and has good scalability during inference. Dai et al. proposed ScanComplete to achieve 3D scan completion and per-voxel semantic labeling. This method leverages the scalability of fully-convolutional neural networks and can adapt to different input data sizes during training and test. A coarse-to-fine strategy is used to hierarchically improve the resolution of the predicted results.

Overall, the volumetric representation naturally preserves the neighborhood structure of 3D point clouds. Its regular data format also allows direct application of standard 3D convolutions. These factors lead to a steady performance improvement in this area. However, the voxelization step inherently introduces discretization artifacts and information loss. Usually, a high resolution leads to high memory and computational costs, while a low resolution introduces loss of details. It is non-trivial to select an appropriate grid resolution in practice.

Sparse Discretization Representation. Volumetric representation is naturally sparse, as the number of non-zero values only accounts for a small percentage. Therefore, it is inefficient to apply dense convolution neural networks on the spatially-sparse data. To this end, Graham et al. proposed submanifold sparse convolutional networks based on the indexing structure. This method significantly reduces memory and computational costs by restricting the output of convolution to be only related to occupied voxels. Meanwhile, its sparse convolution can also control the sparsity of the extracted features. This submanifold sparse convolution is suitable for efficient processing of high-dimensional and spatially-sparse data. Further, Choy et al. proposed a 4D spatio-temporal convolutional neural network called MinkowskiNet for 3D video perception. A generalized sparse convolution is proposed to effectively process high-dimensional data. A trilateral-stationary conditional random field is further applied to enforce consistency.

On the other hand, Su et al. proposed the Sparse Lattice Networks (SPLATNet) based on Bilateral Convolution Layers (BCLs). This method first interpolates a raw point cloud to a permutohedral sparse lattice, BCL is then applied to convolve on occupied parts of the sparsely populated lattice. The filtered output is then interpolated back to the raw point cloud. In addition, this method allows flexible joint processing of multi-view images and point clouds. Further, Rosu et al. proposed LatticeNet to achieve efficient processing of large point clouds. A data-dependent interpolation module called DeformsSlice is also introduced to back project the lattice feature to point clouds.

1.3 Hybrid Methods

To further leverage all available information, several methods have been proposed to learn multi-modal features from 3D scans. Dai and Nießner presented a joint 3D-multi-view network to combine RGB features and geometric features. A 3D CNN stream and several 2D streams are used to extract features, and a differentiable back-projection layer is proposed to jointly fuse the learned 2D embeddings and 3D geometric features. Further, Chiang et al. proposed a unified point-based framework to learn 2D textural appearance, 3D structures and global context features from point clouds. This method directly applies point-based networks to extracts local geometric features and global context from sparsely sampled point sets without any voxelization. Jaritz et al. proposed Multi-view PointNet (MVPNet) to aggregate appearance features from 2D multi-view images and spatial geometric features in the canonical point cloud space.

1.4 Point-based Methods

Point-based networks directly work on irregular point clouds. However, point clouds are orderless and unstructured, making it infeasible to directly apply standard CNNs. To this end, the pioneering work PointNet is proposed to learn per-point features using shared MLPs and global features using symmetrical pooling functions. Based on PointNet, a series of point-based networks have been proposed recently. Overall, these methods can be roughly divided into pointwise MLP methods, point convolution methods, RNN-based methods, and graph-based methods.

Pointwise MLP Methods. These methods usually use shared MLP as the basic unit in their network for its high efficiency. However, point-wise features extracted by shared MLP cannot capture the local geometry in point clouds and the mutual interactions between points . To capture wider context for each point and learn richer local structures, several dedicated networks have been introduced, including methods based on neighboring feature pooling, attention-based aggregation, and local-global feature concatenation.

Neighboring feature pooling: To capture local geometric patterns, these methods learn a feature for each point by aggregating the information from local neighboring points. In particular, PointNet++ groups points hierarchically and progressively learns from larger local regions, as illustrated in Fig. 12(a). Multi-scale grouping and multi-resolution grouping are also proposed to overcome the problems caused by non-uniformity and varying density of point clouds. Later, Jiang et al. proposed a PointSIFT module to achieve orientation encoding and scale awareness. This module stacks and encodes the information from eight spatial orientations through a three-stage ordered convolution. Multi-scale features are concatenated to achieve adaptivity to different scales. Different from the grouping techniques used in PointNet++ (i.e., ball query), Engelmann et al. utilized $K$-means clustering and KNN to separately define two neighborhoods in the world space and feature space. Based on the assumption that points from the same class are expected to be closer in feature space, a pairwise distance loss and a centroid loss are introduced to further regularize feature learning. To model the mutual interactions between different points, Zhao et al. proposed PointWeb to explore the relations between all pairs of points in a local region by densely constructing a locally fully-linked web. An Adaptive Feature Adjustment (AFA) module is proposed to achieve information interchange and feature refinement. This aggregation operation helps the network to learn a discriminative feature representation. Zhang et al. proposed a permutation invariant convolution called Shellconv based on the statistics from concentric spherical shells. This method first queries a set of multi-scale concentric spheres, the max-pooling operation is then used within different shells to summarize the statistics, MLPs and 1D convolution are used to obtain the final convolution output. Hu et al. proposed an efficient and lightweight network called RandLA-Net for large-scale point cloud segmentation. This network utilizes random point sampling to achieve remarkably high efficiency in terms of memory and computation. A local feature aggregation module is further proposed to capture and preserve geometric features.

Attention-based aggregation: To further improve segmentation accuracy, an attention mechanism is introduced to point cloud segmentation. Yang et al. proposed a group shuffle attention to model the relations between points, and presented a permutation-invariant, task-agnostic and differentiable Gumbel Subset Sampling (GSS) to replace the widely used FPS approach. This module is less sensitive to outliers and can select a representative subset of points. To better capture the spatial distribution of a point cloud, Chen et al. proposed a Local Spatial Aware (LSA) layer to learn spatial awareness weights based on the spatial layouts and the local structures of point clouds. Similar to CRF, Zhao et al. proposed an Attention-based Score Refinement (ASR) module to post-process the segmentation results produced by the network. The initial segmentation result is refined by pooling the scores of neighboring points with learned attention weights. This module can be easily integrated into existing deep networks to improve segmentation performance.

Local-global concatenation: Zhao et al. proposed a permutation-invariant PS2-Net to incorporate local structures and global context from point clouds. Edgeconv and NetVLAD are repeatedly stacked to capture the local information and scene-level global features.

Point Convolution Methods. These methods tend to propose effective convolution operators for point clouds. Hua et al. proposed a point-wise convolution operator, where the neighboring points are binned into kernel cells and then convolved with kernel weights. As shown in Fig. 12(b), Wang et al. proposed a network called PCCN based on parametric continuous convolution layers. The kernel function of this layer is parameterized by MLPs and spans the continuous vector space. Thomas et al. proposed a Kernel Point Fully Convolutional Network (KP-FCNN) based on Kernel Point Convolution (KPConv). Specifically, the convolution weights of KPConv are determined by the Euclidean distances to kernel points, and the number of kernel points is not fixed. The positions of the kernel points are formulated as an optimization problem of best coverage in a sphere space. Note that, the radius neighbourhood is used to keep a consistent receptive field, while grid subsampling is used in each layer to achieve high robustness under varying densities of point clouds. In , Engelmann et al. provided rich ablation experiments and visualization results to show the impact of receptive field on the performance of aggregation-based methods. They also proposed a Dilated Point Convolution (DPC) operation to aggregate dilated neighboring features, instead of the K nearest neighbours. This operation is demonstrated to be very effective in increasing the receptive field and can be easily integrated into existing aggregation-based networks.

RNN-based Methods. To capture inherent context features from point clouds, Recurrent Neural Networks (RNN) have also been used for semantic segmentation of point clouds. Based on PointNet , Engelmann et al. first transformed a block of points into multi-scale blocks and grid blocks to obtain input-level context. Then, the block-wise features extracted by PointNet are sequentially fed into Consolidation Units (CU) or Recurrent Consolidation Units (RCU) to obtain output-level context. Experimental results show that incorporating spatial context is important for the improvement of the segmentation performance. Huang et al. proposed a lightweight local dependency modeling module, and utilized a slice pooling layer to convert unordered point feature sets into an ordered sequence of feature vectors. As shown in Fig. 12(c), Ye et al. first proposed a Pointwise Pyramid Pooling (3P) module to capture the coarse-to-fine local structure, and then utilized two-direction hierarchical RNNs to further obtain long-range spatial dependencies. RNN is then applied to achieve an end-to-end learning. However, these methods lose rich geometric features and density distribution from point clouds when aggregating the local neighbourhood features with global structure features . To alleviate the problems caused by the rigid and static pooling operations, Zhao et al. proposed a Dynamic Aggregation Network (DAR-Net) to consider both global scene complexity and local geometric features. The inter-medium features are dynamically aggregated using a self-adapted receptive field and node weights. Liu et al. proposed 3DCNN-DQN-RNN for efficient semantic parsing of large-scale point clouds. This network first learns the spatial distribution and color features using a 3D CNN network, DQN is further used to localize objects belonging to a specific class. The final concatenated feature vector is fed into a residual RNN to obtain the final segmentation results.

Graph-based Methods. To capture the underlying shapes and geometric structures of 3D point clouds, several methods resort to graph networks. As shown in Fig. 12(d), Landrieu et al. represented a point cloud as a set of interconnected simple shapes and superpoints, and used an attributed directed graph (i.e., superpoint graph) to capture the structure and context information. Then, the large-scale point cloud segmentation problem is spilt into three sub-problems, i.e., geometrically homogeneous partition, superpoint embedding, and contextual segmentation. To further improve the partition step, Landrieu and Boussaha proposed a supervised framework to oversegment a point cloud into pure superpoints. This problem is formulated as a deep metric learning problem structured by an adjacency graph. In addition, a graph-structured contrastive loss is also proposed to help the recognition of borders between objects.

To better capture the local geometric relationships in high-dimensional space, Kang et al. proposed a PyramNet based on Graph Embedding Module (GEM) and Pyramid Attention Network (PAN). The GEM module formulates a point cloud as a directed acyclic graph and utilzes a covariance matrix to replace the Euclidean distance for the construction of adjacent similarity matrix. Convolution kernels with four different sizes are used in the PAN module to extract features with different semantic intensities. In , Graph Attention Convolution (GAC) is proposed to selectively learn relevant features from a local neighboring set. This operation is achieved by dynamically assigning attention weights to different neighboring points and feature channels based on their spatial positions and feature differences. GAC can learn to capture discriminative features for segmentation, and has similar characteristics to the commonly used CRF model. Ma et al. proposed a Point Global Context Reasoning (PointGCR) module to capture global contextual information along the channel dimension using an undirected graph representation. PointGCR is a plug-and-play and end-to-end trainable module. It can easily be integrated into an existing segmentation network to achieve performance improvement.

In addition, several very recent work tries to achieve semantic segmentation of point clouds under weak supervision. Wei et al. proposed a two-stage approach to train a segmentation network with subcloud level labels. Xu et al. investigated several inexact supervision schemes for semantic segmentation of point clouds. They also proposed a network that is able to be trained with only partially labeled points (e.g. 10%).

2 Instance Segmentation

Compared to semantic segmentation, instance segmentation is more challenging as it requires more accurate and fine-grained reasoning of points. In particular, it not only needs to distinguish the points with different semantic meanings, but also separate instances with the same semantic meaning. Overall, existing methods can be divided into two groups: proposal-based methods and proposal-free methods. Several milestone methods are illustrated in Fig 13.

These methods convert the instance segmentation problem into two sub-tasks: 3D object detection and instance mask prediction.

Hou et al. proposed a 3D fully-convolutional Semantic Instance Segmentation (3D-SIS) network to achieve semantic instance segmentation on RGB-D scans. This network learns from both color and geometry features. Similar to 3D object detection, a 3D Region Proposal Network (3D-RPN) and a 3D Region of Interesting (3D-RoI) layer are used to predict bounding box locations, object class labels and instance masks. Following the analysis-by-synthesis strategy, Yi et al. proposed a Generative Shape Proposal Network (GSPN) to generate high-objectness 3D proposals. These proposals are further refined by a Region-based PointNet (R-PointNet). The final label is obtained by predicting a per-point binary mask for each class label. Different from direct regression of 3D bounding boxes from point clouds, this method removes a large amount of meaningless proposals by enforcing geometric understanding.

By extending 2D panoptic segmentation to 3D mapping, Narita et al. proposed an online volumetric 3D mapping system to jointly achieve large-scale 3D reconstruction, semantic labeling, and instance segmentation. They first utilized 2D semantic and instance segmentation networks to obtain pixel-wise panoptic labels and then integrated these labels to the volumtric map. A fully-connected CRF is further used to achieve accurate segmentation. This semantic mapping system can achieve high-quality semantic mapping and discriminative object recognition. Yang et al. proposed a single-stage, anchor-free and end-to-end trainable network called 3D-BoNet to achieve instance segmentation on point clouds. This method directly regresses rough 3D bounding boxes for all potential instances, and then utilizes a point-level binary classifier to obtain instance labels. Particularly, the bounding box generation task is formulated as an optimal assignment problem. In addition, a multi-criteria loss function is also proposed to regularize the generated bounding boxes. This method does not need any post-processing and is computationally efficient. Zhang et al. proposed a network for instance segmentation of large-scale outdoor LiDAR point clouds. This method learns a feature representation on the bird’s-eye view of point clouds using self-attention blocks. The final instance labels are obtained based on the predicted horizontal center and the height limits. Shi et al. proposed a hierarchy-aware Variational Denoising Recursive AutoEncoder (VDRAE) to predict the layout of indoor 3D space. The object proposals are iteratively generated and refined by recursive context aggregation and propagation.

Overall, proposal-based methods are intuitive and straightforward, and the instance segmentation results usually have good objectness. However, these methods require multi-stage training and pruning of redundant proposals. Therefore, they are usually time-consuming and computationally expensive.

2.2 Proposal-free Methods

Proposal-free methods do not have an object detection module. Instead, they usually consider instance segmentation as a subsequent clustering step after semantic segmentation. In particular, most existing methods are based on the assumption that points belonging to the same instance should have very similar features. Therefore, these methods mainly focus on discriminative feature learning and point grouping.

In a pioneering work, Wang et al. first introduced a Similarity Group Proposal Network (SGPN). This method first learns a feature and semantic map for each point, and then introduces a similarity matrix to represent the similarity between each paired features. To learn more discriminative features, they use a double-hinge loss to mutually adjust the similarity matrix and semantic segmentation results. Finally, a heuristic and non-maximal suppression method is adopted to merge similar points into instances. Since the construction of a similarity matrix requires large memory consumption, the scalability of this method is limited. Similarly, Liu et al. first leveraged submanifold sparse convolution to predict semantic scores of each voxel and affinity between neighboring voxels. They then introduced a clustering algorithm to group points into instances based on the predicted affinity and the mesh topology. Mo et al. introduced a detection-by-segmentation network in PartNet to achieve instance segmentation. PointNet++ is used as the backbone to predict semantic labels of each point and disjoint instance masks. Further, Liang et al. proposed a structure-aware loss for the learning of discriminative embeddings. This loss considers both the similarity of features and the geometric relations among points. An attention-based graph CNN is further used to adaptively refine the learned features by aggregating different information from neighbors.

Since the semantic category and instance label of a point are usually dependent on each other, several methods have been proposed to couple these two tasks into a single task. Wang et al. integrated these two tasks by introducing an end-to-end and learnable Associatively Segmenting Instances and Semantics (ASIS) module. Experiments show that semantic features and instance features can mutually support each other to achieve an improved performance through this ASIS module. Similarly, Zhao et al. proposed JSNet to achieve both semantic and instance segmentation. Further, Pham et al. first introduced a Multi-Task Point-wise Network (MT-PNet) to assign a label to each point and regularized the embeddings in the feature space by introducing a discriminative loss . They then fused the predicted semantic labels and embeddings to a Multi-Value Conditional Random Field (MV-CRF) model for joint optimization. Finally, mean-field variational inference is used to produce semantic labels and instance labels. Hu et al. first proposed a Dynamic Region Growing (DRG) method to dynamically separate a point cloud into a set of disjoint patches, and then used an unsupervised K-means++ algorithm to group all these patches. Multi-scale patch segmentation is then performed with the guidance of contextual information between patches. Finally, these labeled patches are merged into object level to obtain final semantic and instance labels.

To achieve instance segmentation on full 3D scenes, Elich et al. presented a hybrid 2D-3D network to jointly learn global consistent instance features from a BEV representation and local geometric features of point clouds. The learned features are then combined to achieve semantic and instance segmentation. Note that, rather than heuristic GroupMerging algorithms , a more flexible Meanshift algorithm is used to group these points into instances. Alternatively, multi-task learning is also introduced for instance segmentation. Lahoud et al. learned both the unique feature embedding of each instance and the directional information to estimate the object’s center. Feature embedding loss and directional loss are proposed to adjust the learned feature embeddings in latent feature space. Mean-shift clustering and non-maximum suppression are adopted to group voxels into instances. This method achieves the state-of-the-art performance on the ScanNet benchmark. Besides, the predicted directional information is particularly useful to determine the boundary of instances. Zhang et al. introduced probabilistic embeddings to instance segmentation of point clouds. This method also incorporates uncertainty estimation and proposes a new loss function for the clustering step. Jiang et al. proposed a PointGroup network, which is composed of a semantic segmentation branch and an offset prediction branch. A dual-set clustering algorithm and the ScoreNet is further utilized to achieve better grouping results.

In summary, proposal-free methods do not require computationally expensive region-proposal components. However, the objectness of instance segments grouped by these methods is usually low since these methods do not explicitly detect object boundaries.

3 Part Segmentation

The difficulties for part segmentation of 3D shapes are twofold. First, shape parts with the same semantic label have a large geometric variation and ambiguity. Second, the number of parts in objects with the same semantic meanings may be different.

VoxSegNet is proposed to achieve fine-grained part segmentation on 3D voxelized data under a limited solution. A Spatial Dense Extraction (SDE) module (which consists of stacked atrous residual blocks) is proposed to extract multi-scale discriminative features from sparse volumetric data. The learned features are further re-weighted and fused by progressively applying an Attention Feature Aggregation (AFA) module. Kalogerakis et al. combined FCNs and surface-based CRFs to achieve end-to-end 3D part segmentation. They first generated images from multiple views to achieve optimal surface coverage and fed these images into a 2D network to produce confidence maps. Then, these confidence maps are aggregated by a surface-based CRF, which is responsible for a consistent labeling of the entire scene. Yi et al. introduced a Synchronized Spectral CNN (SyncSpecCNN) to perform convolution on irregular and non-isomorphic shape graphs. A spectral parameterization of dilated convolutional kernels and a spectral transformer network is introduced to solve the problem of multi-scale analysis in parts and information sharing across shapes.

Wang et al. first performed shape segmentation on 3D meshes by introducing Shape Fully Convolutional Networks (SFCN) and taking three low-level geometric features as its input. They then utilized voting-based multi-label graph cuts to further refine the segmentation results. Zhu et al. proposed a weakly-supervised CoSegNet for 3D shape co-segmentation. This network takes a collection of unsegmented 3D point cloud shapes as input, and produces shape part labels by iteratively minimizing a group consistency loss. Similar to CRF, a pre-trained part-refinement network is proposed to further refine and denoise part proposals. Chen et al. proposed a Branched AutoEncoder network (BAE-NET) for unsupervised, one-shot and weakly supervised 3D shape co-segmentation. This method formulates the shape co-segmentation task as a representation learning problem and aims at finding the simplest part representations by minimizing the shape reconstruction loss. Based on the encoder-decoder architecture, each branch of this network can learn a compact representation for a specific part shape. The features learned from each branch and the point coordinate are then fed to the decoder to produce a binary value (which indicates whether the point belongs to this part). This method has good generalization ability and can process large 3D shape collections (up to 5000+ shapes). However, it is sensitive to initial parameters and does not incorporate shape semantics into the network, which hinders this method to obtain a robust and stable estimation in each iteration. Yu et al. proposed a top-down recursive part decomposition network (PartNet) for hierarchical shape segmentation. Different from existing methods that segment a shape to a fixed label set, this network formulates part segmentation as a problem of cascade binary labeling, and decompose the input point cloud to an arbitrary number of parts based on the geometric structure. Luo et al. introduced a learning-based grouping framework for the task of zero-shot 3D part segmentation. To improve the cross-category generalization ability, this method tends to learn a grouping policy that restricts the network to learn part-level features within the part local context.

4 Summary

Table V shows the results achieved by existing methods on public benchmark, including S3DIS , Semantic3D , ScanNet , and SemanticKITTI . The following issues need to be further investigated:

Thanks to the regular data representation, both projection-based methods and discretization-based methods can leverage the mature network architecture from their 2D image counterparts. However, the main limitation of projection-based methods lies in the information loss caused by 3D-2D projection, while the main bottleneck for discretization-based methods is the cubically increased computational and memory costs caused by the increase of the resolution. To this end, sparse convolution building upon indexing structures would be a feasible solution and worth further exploration.

Point-based networks are the most frequently investigated methods. However, point representation naturally does not have explicit neighboring information, most existing point-based methods resort to expensive neighbor searching mechanisms (e.g., KNN or ball query ). This inherently limits the efficiency of these methods, the recently proposed point-voxel joint representation would be an interesting direction for further investigation.

Learning from imbalanced data is still a challenging problem in point cloud segmentation. Although several approaches have achieved a remarkable overall performance, their performance on minority classes is still limited. For example, RandLA-Net achieves an overall IoU of 76.0% on the reduced-8 subset of Semantic3D, but a very low IOU of 41.1% on the class of hardscape.

The majority of existing approaches work on small point clouds (e.g., 1m$\times$1m with 4096 points). In practice, the point clouds acquired by depth sensors are usually immense and large-scale. Therefore, it is desirable to further investigate the problem of efficient segmentation of large-scale point clouds.

A handful of works have started to learn spatio-temporal information from dynamic point clouds. It is expected that the spatio-temporal information can help to improve the performance of subsequent tasks such as 3D object recognition, segmentation, and completion.

Conclusion

This paper has presented a contemporary survey of the state-of-the-art methods for 3D understanding, including 3D shape classification, 3D object detection and tracking, and 3D scene and object segmentation. A comprehensive taxonomy and performance comparison of these methods have been presented. Merits and demerits of various methods are also covered, with potential research directions being listed.

Acknowledgments

This work was partially supported by the National Natural Science Foundation of China (No. 61972435, 61602499, 61872379), the Natural Science Foundation of Guangdong Province (2019A1515011271), the Science and Technology Innovation Committee of Shenzhen Municipality (JCYJ20190807152209394), the Australian Research Council (Grants DP150100294 and DP150104251), the China Scholarship Council (CSC) and the Academy of Finland.

References