Hierarchical Point-Edge Interaction Network for Point Cloud Semantic Segmentation

Introduction

With increasing capability of 3D sensing hardware, it is now easy to capture 3D data in many scenarios. Compared with 2D images, 3D data provides richer information about the environment. 3D data is in general view-independent and captures 3D structure, making it possible to incorporate geometry information in scene understanding tasks.

Learning-based approaches were proposed to solve various 3D vision problems, e.g., shape classification, scene semantic/instance segmentation, and 3D object detection. Unlike 2D images, in which pixel grids are regular with object color information, 3D object data scatters, with most space actually not occupied. Therefore, directly voxelizing 3D scenes and extending deep neural network operations from 2D to 3D is inefficient. Several voxel-based methods, such as Submanifold Sparse Convolution and O-CNN , improve the 3D convolution efficiency. However, since voxelization is accompanied by loss of information, high-resolution 3D models are needed to uphold the data precision, even though it unavoidably costs large memory and computation resource.

From another perspective, PointNet directly processes 3D points in a network, only considering regions covered by the 3D points. PointNet++ further adopts a hierarchical encoder-decoder structure to consider local regions, which downsamples point clouds in layers first and gradually interpolates them to the original resolution. This framework just utilizes weak connection between each point and its local context, since point features are extracted independently by the multi-layer perceptrons (MLP). In segmentation tasks, it is commonly known that local context is crucial for labeling the semantic categories. This motivates us to further explore the semantic relation between points and their local contextual neighbors to extract more discriminative features for 3D semantic scene labeling.

To explore the semantic relation between points in a local region and utilize the contextual information, we explicitly build edges between points and their contextual neighbors and establish a hierarchical edge branch with an auxiliary edge loss, as shown in Figure 1.

Specifically, besides the encoder-decoder point branch as in PointNet++, our new edge branch accepts point features from different layers and progressively produces edge features, which are then fed to point branch for fusing information in local graphs. For each point, the corresponding edge features provide local intrinsic geometric and regional semantic information to enhance point representation.

Instead of building isolated graphs for points in each layer, we design a hierarchical graph construction process to gradually take point features at different layers into the edge branch. Edge features of adjacent layers are connected by an operator, named “edge upsample”. Consequently, edges on full-resolution point cloud encode multi-layer features, providing comprehensive data for final prediction.

We regularize the final edge features considering semantic consistency of the two connected points, which helps increase the discrimination ability between inter- and intra-category feature pairs, implicitly pulling points with the same semantic label closer in the feature space.

The decent performance of our method compared with all existing point-based neural networks on the large-scale scene labeling datasets, i.e., Stanford Large Scale 3D Indoor Space (S3DIS) and ScanNet , manifests the effectiveness of our framework.

Related Work

To process 3D data, one typical approach is to store the data in volume grids and adopt 3D convolutions . Since most voxels are unoccupied, Submanifold Sparse Convolution Network defines a sparse convolution operation to process spatially-sparse 3D data. OctNet , on the other hand, represents the data using unbalanced octrees and defines network operations on these octrees to enable deeper neural networks without sacrificing the precision. Similarly, O-CNN uses an octree to enable 3D CNN on high-resolution 3D data.

Another approach is to use multi-view 2D images, to which 2D convolutions can be directly applied. However, these approaches overlook the geometric structure in objects and scenes, especially the view-occluded 3D structures. Other methods consider 3D object surface and apply convolutions on it for semantic analysis.

2 Point-based Deep Neural Network

PointNet is the first deep neural network to directly process 3D point coordinates, with MLPs and max-pooling for extracting features. Since max-pooling is a global operation on all the points, PointNet lacks local region understanding. PointNet++ further applies a hierarchical structure and uses $k$-NN followed by max-pooling to capture regional information. Since it aggregates local features simply via a max-pooling, regional information is not yet fully utilized.

Recently, much effort has been made for effective local feature aggregation. SPLATNet maps points into a high-dimensional sparse lattice and performs convolution on it. RSNet projects features of unordered points into an ordered sequence of feature vectors and applies Recurrent Neural Network layers to model local dependency. PointCNN explores convolution on point clouds and addresses the point ordering issue by permuting and weighting input points and features with the $\mathcal{X}$-Conv operator. Besides, methods of explore local context based on graphs.

ECC organizes point clouds as graphs and uses graph convolutions to dynamically learn weights to combine local features. DGCNN proposes the EdgeConv module to generate edge features that describe the connection between a point and its nearest neighbors. PointWeb further connects every point pairs in a local region to obtain more representative region features. KCNet creates $k$-nearest neighbor graphs and applies kernel correlation to learn local structures over point neighborhood. PCCN and PointConv connect each point with its $k$-nearest neighbors and extend the convolution operation from regular grids to irregular point clouds by adaptively projecting the relative position of two points to a convolution weight. Compared to PCCN, PointConv additionally considers point distribution density. Spectral Graph Convolution performs graph convolution after a graph Fourier transform. Superpoint Graph (SPG) splits the point cloud into geometrically-homogeneous partitions and builds a super-point graph, followed by a graph neural network to produce semantic labels.

In our work, we also propose a graph for point cloud processing, and yet focus particularly on exploring the semantic relation between points and their contextual neighbors for semantic segmentation through explicit edges. The key distinction of our method from other graph-based frameworks is that instead of fixing the graph and point resolution (e.g., PCCN and KCNet ) or building independent graphs at each scale (e.g., PointConv , PointWeb and ECC ), our graph is hierarchically constructed. We construct an edge branch, in which we fuse multi-scale point features and propagate edge features over multiple scales to enable longer distances of message passing hierarchically over edges without large memory overhead. Moreover, we propose edge loss aiming to encode the edges with exact semantic consistency information and increase the discrimination power among point features with different categories.

With meaningful edge features, we further feed edge features into each scale of the point branch to offer contextual information. To pass messages via edges, PointConv and PCCN adaptively learn weights from edges to fuse point features, while KCNet defines a point-set kernel and kernel correlation to aggregate local features along edges. Different from these methods, our approach concatenates each point feature with the max-pooled corresponding edge features. Our approach requires less parameters to learn and preserves the distinctiveness of individual point features (Section 4.4 provides more discussions).

Our Approach

We design a hierarchical edge branch collaborating with the point prediction branch for point cloud semantic segmentation, as shown in Fig. 2. We progressively enlarge the graph, upsample edge features, and accept point features in different layers to refine the edge features. Edge features in different layers then provide extra contextual information for point feature learning. The final edge features are regularized with semantic consistency of their two-end points, which serve as auxiliary supervision for point features.

In this section, we first introduce the new edge branch, covering especially the interaction between point and edge branches, in Section 3.1. Then the hierarchical graph construction framework, which enables integration of different-layer information for edge prediction is described in Section 3.2. Section 3.3 depicts the loss regularizing both category prediction of each point and semantic-consistency prediction of each edge.

Given a point cloud with $N$ points $\mathcal{P}=\{p_{1},p_{2},...,p_{N}\}$, we construct a directed graph $G=(V,E)$, where $V=\mathcal{P}$ and $E$ includes the edges that connect each point to its contextual points. Here, $G$ is hierarchically constructed in a coarse to fine manner. We denote the graph in layer $L$ as $G_{L}$. A larger $L$ indicates a layer with more points, and layer is the coarsest layer with the least points. The detailed graph construction process is depicted later in Section 3.2. Here, we first introduce the constitution of edge branch and how it interacts with the point branch.

As shown in Fig. 2, for the point branch, we follow PointNet++ to create a hierarchical encoder-decoder structure with previous features in point encoder connected to the corresponding point decoder layers through skip-connection, thus passing detailed low-level information. The point cloud is downsampled and then upsampled in the process. Meanwhile, we construct an edge branch with consecutive edge modules, taking both features from the corresponding point module and the previous edge module.

The procedure is to extract edge features from the coarsest layer to grab high-level information with the largest receptive field, and progressively fuse point features from finer layers into edges, in parallel with the point decoding stage. Point features from the encoder layers are also used in the process, along with skip-connection to the corresponding decoder layers.

Although both abstract global features from the coarser layers and detailed information from finer layers are important, the most essential data for edge prediction is from the last layer with the most refined point features. With this consideration, edge features are encoded in a coarse-to-fine manner, making point features in the finest layer fused at last. The hierarchical edge features are also fed to the corresponding point modules to provide additional contextual information.

where $M_{encoder}$ denotes the edge encoder and $M_{upsample}$ is the edge upsampling module, which maps edge features in graph $G_{L-1}$ to graph $G_{L}$. The graph construction and edge upsampling process will be described in Section 3.2.

For each edge $e_{i,j}=(p_{i},p_{j})\in E_{L}$, its edge feature at layer $L$ is written as

where $F_{i}^{L}$ and $F_{j}^{L}$ are the point features of $p_{i}$ and $p_{j}$, respectively. $H_{i,j}^{L-1\to L}$ is the edge feature upsampled from layer $L-1$ to layer $L$.

As illustrated in Fig. 3(b), $M_{encoder}$ for a single edge can be expanded as

where $[\cdot,\cdot,\cdot]$ concatenates the three elements, and $p_{i},p_{j}$ here represent 3D point coordinates. The two point features are concatenated for completely preserving information of the two points. Also, we provide $(p_{j}-p_{i})$ to indicate the relative position between the two points. Other implementations of $f_{edge}$ are discussed in the experiment part.

1.2 Incorporation of Edges in Point Prediction

For layer $L$, every point in graph $G_{L}$ links to other contextual points. So corresponding edges are expected to pass the contextual information back to the point. To this end, the edge features with respect to point $p_{i}$ are operated by max-pooling as a region guidance. Let $E_{L}(p_{i})$ denote the set containing all edges starting from $p_{i}$, the corresponding set of edge features is

The point feature $F_{i}^{L}$ is then updated by

Fig. 4 gives an illustration of the process.

By incorporating edge information in point features, we enlarge the message passing range. The local region feature provided by the edges allows the point feature extractor to see farther in each layer. Additional contextual information including intrinsic geometry and semantic relation in the local region is incorporated in the region feature to benefit segmentation. We experiment with other schemes for message passing. Section 4.4 gives more discussions.

By helping feature extraction in the other branch, point and edge features become more powerful in final prediction.

2 Hierarchical Graph Construction

Instead of building graphs separately at each layer, we build the graph hierarchically, as shown in Fig. 5. By designing the “edge upsample” operation with each edge aware of associated edges in previous layer, we enlarge the receptive field and enable longer-range message passing for edges.

As shown in Fig. 5, the graph is initialized in the coarsest layer (layer 0). The initial graph $G_{0}$ is constructed by connecting each point with its nearest $k_{0}$ points. Mathematically, $G_{0}=(V_{0},E_{0})$ is formulated as

where $\mathcal{P}_{0}$ is the point set in layer 0, which is downsampled from the original point set with farthest point sampling (FPS) in encoding layers. $N_{k_{0}}(p_{i})$ is the set of the $k_{0}$-nearest neighbors of point $p_{i}$, including itself.

2.2 Hierarchical Architecture

Along with the decoding process of point features, we gradually enlarge the graph and enrich the edge features with more details. The process is illustrated in Fig. 5.

Consider two adjacent layers $L-1$ and $L$ with vertices $V_{L-1}$ and $V_{L}$ as the point set in that layer, respectively. The graph $G_{L}$ is constructed by first finding the $k_{L}$ nearest neighbors for each point in $V_{L}$. Let $G_{L}^{(0)}=(V_{L},E_{L}^{(0)})$ denote such initial $L$-layer graph. For each edge $e_{i,j}=(p_{i},p_{j})\in E_{L}^{(0)}$, we consider the set consisting of possible neighboring edges in layer $L-1$ as

where $N_{k}^{L-1}(p_{i})\subseteq V_{L-1}$ is the $k$-nearest neighbors of $p_{i}\in V_{L}$ in layer $L-1$. $p_{i}$ is included in $N_{k}^{L-1}(p_{i})$ if $p_{i}\in V_{L-1}$.

We then check whether edges in $E_{ne}^{L-1}(e_{i,j})$ exist in $E_{L-1}$ – the edge set of $G_{L-1}$. If edge $e_{i,j}$ connects two distant points, for which even in the coarser layer $L-1$ there is no connection between the two corresponding regions, we do not take the edge into consideration in layer $L$. Hence, if $E_{ne}^{L-1}(e_{i,j})\cap E_{L-1}=\O$, edge $e_{i,j}$ is discarded from $E_{L}^{(0)}$. Following this principle, the final graph $G_{L}=(V_{L},E_{L})$ has an edge set of

where $E_{L}(p_{i})$ (edges starting from $p_{i}$) is expressed as

Note that at least $e_{i,i}$ is reserved in $E_{L}(p_{i})$ in some extreme cases.

In PointNet++ , point feature of $p_{i}$ in layer $L$ is propagated from layer $L-1$ by interpolating feature values of its $k$ nearest neighbors in layer $L-1$ as

We similarly propagate edge features in layer $L-1$ to layer $L$ as

The interpolation weights are based on the inverse distance of the two pairs of end points. For $H_{i^{\prime},j^{\prime}}^{L-1}$, the weight is formulated as

where $p_{i^{\prime}},p_{j^{\prime}}\in V_{L-1}$, $p_{i},p_{j}\in V_{L}$ represent point coordinates, $\epsilon=1e-8$ and $t$ is set to 2. The weights are then normalized as

3 Loss Function

We optimize the point and edge branches jointly with the combined loss on the two branches as

where $\lambda_{1}$ and $\lambda_{2}$ adjust the ratio of the two losses.

The final point features are followed by an MLP to produce point-wise semantic predictions. We further use the final edge predictions as weights to aggregate point scores and get refined point predictions. Cross entropy loss is applied to constrain the point predictions.

The edge features in the final graph $G$ are regularized by the edge labels, which represent whether the two-end points of the edge are in the same category or not. The label for edge $e_{i,j}=(p_{i},p_{j})\in E$ is set as

where $l_{i}^{p}$ and $l_{j}^{p}$ are the point semantic labels of $p_{i}$ and $p_{j}$. An MLP is adopted to produce the per-edge prediction. Binary cross entropy loss is chosen for the edge loss as

where $pred_{i,j}^{e}$ is the edge prediction for $e_{i,j}$, and $\alpha$ balances the two kinds of edges, as there are more intra-class edges than inter-class ones considering the local neighborhood.

The final edge feature for each edge can be deemed as a function on features of the two regions centered at the two-end points. Information from different layers are taken into account. More details are preserved by encoding at last. Hence, the edge loss guides the edge encoder to seek difference between the intra- and inter-class feature pairs, and implicitly serves as auxiliary supervision for point features. It increases the discrimination power among point features in different categories. Also, with the edge supervision, more exact contextual information is passed to points via edges to enhance point features.

Experiments

We conducted experiments on two representative and challenging large-scale scene labeling datasets, i.e., S3DIS and ScanNet v2 , with ablation analysis presented on the ScanNet v2 val set and S3DIS Area 5.

The point branch contains an encoder with four down-sampling layers and a decoder with four upsampling layers. The numbers of points, $N_{0},N_{1},N_{2},N_{3},N_{4}=N$, in the decoder are 16, 64, 256, 1,024, and 4,096, respectively. The edge branch has five blocks with $k$ (number of nearest neighbors) set to $4,6,10,14,16$ from layer 0 to 4. $k$ is chosen as 3 for point and edge feature interpolation.

The whole network was trained in an end-to-end manner using the SGD optimizer with batch size 16 and base learning rate 0.05. For S3DIS, we train the network for 100 epochs and decay the rate by 0.1 for every 25 epochs. For ScanNet, we train the network for 120 epochs and decay the rate by 0.1 for every 30 epochs. The momentum and weight decay are set to 0.9 and 0.0001 respectively.

2 Datasets

The dataset has 6 areas with a total of 271 rooms. Each room is provided as points with RGB information. Each point has a semantic label from 13 categories of floor, window, door, etc. In each training iteration, we randomly sample blocks in the training areas, with 4,096 points randomly selected per block. We set the block size as $0.8m\times 0.8m$ with $0.1m$ padding. Also, we represent each point as a 9D vector with $XYZ$, $RGB$, and normalized position in room. All points in the test areas are used in evaluation. Two settings are adopted : (i) splitting Area 5 as the test set and using others for training; and (ii) adopting 6-fold cross validation, with each of the 6 areas taking as the test set once.

The dataset has 1,613 scans with a train/validation/test split of 1,201/312/100. Excluding the ‘unannotated’ points, each point in the scans has a label from 20 categories of wall, shower curtain, etc. To prepare the input data, we follow previous work to randomly sample blocks in rooms and sample 4,096 points per block. Again, we use $0.8m\times 0.8m$ block size and $0.1m$ padding. Here, each input point feature is a 6D vector ($XYZ$ & RGB). We evaluated on both the validation and test sets. Since the semantic annotation for the test sets is not publicly available, we submitted our predictions to the official server to obtain the evaluation results.

It includes the class-wise mean of intersection over union (mIoU), class-wise mean of accuracy (mAcc) and point-wise overall accuracy (OA).

3 Main Results

Table 1 lists quantitative results of different methods on S3DIS Area 5. Compared to previous approaches, ours yields the highest scores in terms of all the three metrics. Specifically, our model yields mIoU 61.85%, exceeding the former best by 3.58%. Table 2 shows the comparison among different architectures on 6-fold cross validation. Ours also reaches the first place for all the three items.

Table 3 lists results of our framework and other point-based methods on ScanNet v2 test set. All methods use only point clouds with RGB color as input without voxelization. Our approach outperforms others by a large margin: 6.2% higher in absolute mIoU and 11.2% better relatively. Visual results are shown in Figs. 7 and 8. Our method segments objects even in complex scenes. It is notable that several detailed structures are classified and segmented from the surroundings, manifesting the effectiveness of our method.

4 Ablation Study

For ScanNet v2, the models are trained on training set and evaluated on validation set. For S3DIS, the models are trained on Areas 1-4 & 6 and evaluated on Area 5.

We explore different ways of incorporating point information into edges, including Subtraction, Summation, Hadamard product, ‘ConcatSub’, and Concatenation. Here ‘ConcatSub’ is defined as

Table 4 shows comparison of the results. Overall, concatenation yields the best result due to preservation of most point information. Summation, Subtraction, and Hadamard Product all cause information loss in the level of point features. ‘ConcatSub’ achieves similar performance with Concatenation, since the two-point features can be restored in this type of operations.

Besides the approach described in Section 3.1.2, we also experimented with another scheme which is inspired by graph convolution , where the edge features are further encoded to form weights for the linked points. The point features are then updated as a weighted sum of the adjacent point features. We denote this scheme as adaptive aggregation (AdaAggre) and test the two settings, with and without softmax, for the weights. Table 5 lists the experimental results on ScanNet v2 validation set.

The performance gain for the graph-convolution-style methods is lower than max-pooling followed by concatenation. It may be because during the point decoding, it is not very helpful to mix point features in each local neighborhood. Instead, the combined contextual feature reveals the relation of a point with its neighborhood. It can better preserve the point’s own distinctiveness.

We build connection between edge features of adjacent layers by “edge upsample”. We also experimented on ScanNet dataset with removing hierarchical graph construction and building the graph of each layer separately without edge upsampling.

The mIoU/mAcc/OA (%) results are 57.01/66.52/83.57 respectively, much lower than our full framework with 63.36/72.61/86.13. The connected edge branch optimally incorporates the point features in different layers, enabling effective learning for the edge features.

Conclusion

We have designed a hierarchical point-edge interaction network, in which an edge branch is proposed to work with the encoder-decoder point branch for point cloud semantic segmentation. The proposed hierarchical graph framework enables the edge branch to progressively integrate different-layer point features. Also, the generated edge features are incorporated into the point branch to provide contextual information. The final edge features are supervised by the semantic consistency of related points to implicitly regularize the point features. All these steps make semantic relationship with local context well utilized via edges.

With the high-quality point prediction results and generality of the framework applicable to different datasets, we believe the proposed method will broadly benefit 3D understanding in the community. In the future, we will explore multi-range edge construction to gather both close-range and long-distance contextual information.

This project is supported in part by the Research Grants Council of the Hong Kong Special Administrative Region (CUHK 14203416 & 14201918).

References