Fast Point Transformer

Introduction

3D scene understanding is a fundamental task due to its importance to various fields, such as robotics, intelligent agents, and AR/VR. Recent approaches utilize the deep learning frameworks, but processing a large-scale 3D scene as a whole remains a challenging problem because it involves extensive computation and memory budgets. As an alternative, some methods crop 3D scenes and stitch predictions , or others approximate point coordinates for efficiency . Such techniques, however, typically lead to a substantial increase of inference time and/or degrade the final output due to the local or approximate predictions. Achieving both fast inference time and high accuracy is thus one of the primary challenges in the 3D scene understanding tasks.

The pioneering 3D understanding approaches, PointNet and PointNet++ process point clouds with multi-layer perceptrons (MLPs), which preserve permutation-invariance of the point clouds. Such point-based methods introduce impressive results recently, and Point Transformer shows superior accuracy based on the local self-attention mechanism. However, it involves manual grouping of point clouds using $k$ nearest neighbor search. Furthermore, scene-level inference with the point-based methods typically requires dividing a large-scale scene into smaller regions and stitching the predictions on them. While Voxel-based methods are alternatives for a large-scale 3D scene understanding due to their effectiveness of the network design, they may lose fine geometric patterns due to quantization artifacts. Hybrid methods reduce the quantization artifacts by utilizing both point-level and voxel-level features. However, approaches in this category require additional memory space to cache both features.

We propose Fast Point Transformer, which effectively encodes continuous positional information of large-scale point clouds. Our approach leverages local self-attention of point clouds with voxel hashing architecture. To achieve higher accuracy, we present centroid-aware voxelization and devoxelization techniques that preserve the embedding of continuous coordinates. The proposed approach reduces quantization artifacts and allows the coherency of dense predictions regardless of rigid transformations. We also introduce a reformulation of the standard local self-attention equation to reduce space complexity further. The proposed local self-attention module can replace the convolutional layers for 3D scene understanding. Based on this, we introduce a local self-attention based U-shaped network, which naturally builds a feature hierarchy without manual grouping of point clouds. As the result, Fast Point Transformer collects rich geometric representations and exhibits a fast inference time even for large-scale scenes.

We conduct experiments using two datasets of large-scale scenes: S3DIS and ScanNet . Our method shows competitive accuracy in the semantic segmentation task on various voxel hashing configurations. We also apply the Fast Point Transformer network as a backbone of VoteNet to show the applicability in the 3D object detection task. We use ScanNet dataset for the 3D detection, and our model shows better accuracy (mAP) than other baselines that use point- or voxel-based network backbones. Besides, we introduce a novel consistency score metric, named $\operatorname{CScore}$, and demonstrate that our model outputs more coherent predictions under rigid transformations.

In summary, our contributions are as follows:

We propose a novel local self-attention-based network, called Fast Point Transformer that can handle large-scale 3D scenes quickly.

We introduce a lightweight local self-attention module that effectively learns continuous positional information of 3D point clouds while reducing space complexity.

We show that our model produces significantly more coherent predictions than the previous voxel-based approaches using the proposed evaluation metric.

We demonstrate fast inference of our voxel-hashing-based architecture; our network performs a 129 times faster inference than Point Transformer does, obtaining a reasonable accuracy trade-off in 3D semantic segmentation on S3DIS dataset .

Related Work

In this section, we review point-based, voxel-based, and hybrid methods for 3D scene understanding and then revisit the attention-based models.

Point-based methods. PointNet introduces a multi-layer perceptrons (MLP) based approach for understanding 3D scenes. PointNet++ advances the PointNet by adding hierarchical sampling strategies. Recent studies attempt to apply convolution on point clouds since the heuristic local sampling and grouping mechanisms used in PointNet++ can be represented by the convolution. However, applying convolution on point clouds is challenging since 3D points are sparse and unordered. KPConv mimics convolution using kernel points defined in the continuous space. They construct a $k$-d tree to perform point-wise convolution on the query points within a certain radius at the inference stage in exchange for inefficiency at the data preprocessing stage. Mao et al. adopt discretized convolution kernels instead of continuous kernels for efficiency and perform convolution on every point in a point cloud, which poses a bottleneck when processing large-scale 3D scene point clouds. More recently, Guo et al. and Zhao et al. utilize local self-attention operations to learn richer feature representations than the fixed kernel-based methods . In fact, most point-based methods adopt expensive operations, such as $k$ nearest neighbor search or $k$-d tree construction, resulting in heavy computational overhead when processing large-scale 3D scenes.

Voxel-based methods. Sparse convolution constructs fully convolutional neural networks using discrete sparse tensors for fast processing of voxel data. The sparse convolution performs convolution on all valid neighbor voxels that are efficiently found using a hash table with constant time complexity, i.e., $\mathcal{O}(1)$. Mao et al. propose a voxel-based transformer architecture that adopts both local and dilated attention to enlarge receptive fields of the model. Despite the effectiveness of voxel-based work on large-scale point clouds, they often fail to capture fine patterns of point clouds due to the quantization artifacts produced during voxelization. In other words, the features extracted by voxel-based methods are inconsistent with respect to the voxel size .

Hybrid methods. Another approach to handle point clouds is to extract both point- and voxel-level features. Recent work attaches point-based layers, e.g., mini-PointNet, on top of the voxel-based methods to relieve the quantization artifacts produced during voxelization. They take advantage of fast neighbor search of voxel-based methods and high capability of capturing fine-geometries of point-based methods. However, the hybrid methods suffer from larger computation and memory budgets since these approaches store both point- and voxel-level features.

Attention-based networks. Discussions regarding the attention operation have dominated research in recent years in natural language processing . Moreover, recent vision work has attempted to exploit the advantages of attention-based models. Prior research generally confirms that global self-attention is infeasible to be adopted in 3D vision tasks due to its costly operations. Thus, recent work widely utilizes local self-attention to process 3D point clouds. Guo et al. and Zhao et al. handle irregularity of point clouds with k nearest neighbor search, resulting in a remarkable performance gain.

Fast Point Transformer

Fast Point Transformer processes the point cloud through three steps: (Step 1) Centroid-aware voxelization, (Step 2) Lightweight self-attention, and (Step 3) Centroid-aware devoxelization. Figure 2 shows the overall architecture.

(Step 2) The lightweight self-attention (LSA) block takes $\mathcal{V}=\{({\mathbf{v}}_{i},{\mathbf{f}}_{i},{\mathbf{c}}_{i})\}_{i=1}^{I}$ and updates the feature ${\mathbf{f}}_{i}$ to the output feature ${\mathbf{f}}^{\prime}_{i}$ using local self-attention. In this procedure, querying neighbor voxels can be done with voxel hashing having $\mathcal{O}(1)$ complexity for a single query.

2 Centroid-aware Voxel & Devoxelization

where $\oplus$ denotes vector concatenation and $\Omega$ is a permutation-invariant operator, e.g., $\operatorname{average}(\cdot)$.

We state that some voxel-based methods introduce barycentric interpolation to embed ${\mathbf{f}}_{i}$ into regular grids ${\mathbf{v}}_{i}$ for voxelization. The proposed centroid-aware voxelization is different from those methods in that it encodes the centroid-to-point position into ${\mathbf{f}}_{i}$ at continuous centroid coordinate ${\mathbf{c}}_{i}$. The proposed centroid-aware voxeliztion is also different from other class of voxel-based methods that apply average- or max-pool voxel features without using intra-voxel coordinates of points.

3 Lightweight Self-Attention

Although the voxel hashing enables a fast neighbor search with time complexity of $\mathcal{O}(1)$ for a single query, designing a memory-efficient form of continuous positional encoding $\delta({\mathbf{c}}_{i},{\mathbf{c}}_{j})$ still remains a challenging problem. Specifically, inspired by $\operatorname{MLP}({\mathbf{p}}_{i}-{\mathbf{p}}_{j})$ in Point Transformer , implementing $\delta({\mathbf{c}}_{i},{\mathbf{c}}_{j})$ as $\operatorname{MLP}({\mathbf{c}}_{i}-{\mathbf{c}}_{j})$ requires $\mathcal{O}(IKD)$ space complexity, where $K$ is the cardinality of neighboring voxels. This is because there can be $\mathcal{O}(IK)$ different relative positions of $({\mathbf{c}}_{i}-{\mathbf{c}}_{j})$ for possible $(i,j)$ pairs due to the continuity of ${\mathbf{c}}$ as shown in Figure 3.

Reducing space complexity. We introduce a coordinate decomposition approach to reduce space complexity. Given a query voxel $({\mathbf{v}}_{i},{\mathbf{f}}_{i},{\mathbf{c}}_{i})$ and a key voxel $({\mathbf{v}}_{j},{\mathbf{f}}_{j},{\mathbf{c}}_{j})$, the relative position of centroids ${\mathbf{c}}_{i}-{\mathbf{c}}_{j}$ can be decomposed as

We illustrate the reduction of the space complexity in Figure 3, and evaluate the effectiveness of the decomposition in Table A4 and Table A5 of the supplementary material.

Lightweight self-attention layer. Now, we propose the new local self-attention layer, named LSA layer, by defining attention function $a(\cdot)$ in Eq. (6) as

4 Network Architecture

We develop Fast Point Transformer for dense prediction on point cloud based on the modules introduced above. Using coordinate hashing (Sec. 3.2) and decomposed positional encodings (Sec. 3.3), Fast Point Transformer is less prone to quantization errors than previous voxel-based methods , while also being significantly faster than point-based methods in terms of both space and time. Furthermore, the proposed local self-attention layer can be easily be integrated to voxel-based downsampling and upsampling layer without introducing heuristic sampling and grouping mechanisms that are often used in the point-based methods . Note that we can build local self-attention networks by substituting convolution layers with LSA layers. Therefore, any sparse CNN architecture can be modified to faciliate local self-attention, e.g., ResNet and U-Net . We implement our model for semantic segmentation using the U-Net architecture. Further details are described in the supplementary material.

Experiments

In this section, we evaluate our model on two popular large-scale 3D scene datasets: S3DIS and ScanNet . We have selected the two datasets due to their rich diversity and densely annotated labels. We first validate the robustness of our approach to voxel hashing configurations described in Sec. 4.3. Then, we compare the proposed method with the state of the art and discuss the results in Sec. 4.4 and Sec. 4.5. Specifically, we provide stochastic numbers averaged from three different experiments with the same training configuration except random seed numbers for the comparison tables: Table 1, Table 2, Table 3, Table 4, and Table 8.

S3DIS is a large-scale indoor dataset which consists of six large-scale areas with 271 room scenes. We test on Area 5 and utilize the other splits during training. Following , we do not use any preprocessing methods, e.g., cropping into small blocks, that are widely used in point-based methods .

ScanNet. We use the second official release of ScanNet , which consists of 1.5k room scenes with some rooms captured repeatedly with different sensors. Following the experimental settings of prior work , our model uses point-wise RGB colors as input point features $\{\mathbf{i}_{n}\}$ both for 3D semantic segmentation task and 3D objection detection.

2 Baselines

We have selected PointNet , PointWeb , SPGraph , PointConv , PointASNL , KPConv , PAConv , Point Transformer , SparseConvNet , and MinkowskiNet as the baseline approaches. MinkowskiNet32 and MinkowskiNet42 are compared as representative voxel-based methods that comprise 32 and 42 U-Net layers, respectively. We reproduce MinkowskiNet42 with the official source code and denote it as MinkowskiNet42†, with different voxel sizes. PointNet , SPGraph , PointWeb , KPConv , PAConv and Point Transformer are selected since they are representative point-based methods. The main difference between KPConv and the others is that KPConv uses a $k$-d tree to boost its inference time while the others do not. We follow the official guideline of the methods and reproduce the results. A more recent method, Point Transformer has also been selected due to its superiority on several datasets. Unlike our method and selected baselines, other approaches use additional inputs, e.g., 2D images or meshes. Accordingly, we have excluded these methods from the comparison.

3 Consistency Test

4 3D Semantic Segmentation

We compare our approach with the state of the art in 3D semantic segmentation on S3DIS and ScanNet . We use the mean of class-wise IoU scores as the primary evaluation metric for both datasets.

Table 2 theoretically analyzes the time complexity and reports the average wall-time latency of each method when processing S3DIS Area 5 scenes. We measure the inference time of MinkowskiNet42†, PointNet , SPGraph , PointWeb , KPConv , PAConv , and Point Transformer using the official codes. We use the same machine with Intel(R) Core(TM) i7-5930K CPU and a single NVIDIA Geforce RTX 3090 GPU to measure the latency of methods. Detailed information about the time complexity analysis is included in the supplementary material.

Due to the preprocessing stage and stitching the multiple local predictions or multiple inferences , the point-based methods take much more time to inference a single scene than our approach. Note that KPConv constructs $k$-d tree, but we do not include this process into inference time. Our Fast Point Transformer processes a large-scale scene at least 83 times faster than point-based methods as shown in Table 2. Specifically, PointNet takes 18.16 seconds for processing a scene on average because it crops the scene into $1m\times 1m\times 1m$ blocks, predicts on the blocks, and stitches the predictions for the scene-level prediction (denoted by ‘Crop-and-stitch’ in Table 2). Moreover, Fast Point Transformer outperforms MinkowskiNet42† by 1.4 absolute percentage score in mean IoU ($\%$) with a comparable speed. Given the reported results by Zhao et al. , Point Transformer shows the best accuracy. However, Point Transformer shows 129 times slower inference speed than our approach. This is because it grid-subsamples points and inferences the sampled points multiple times with the expensive $k$ nearest neighbor search to cover the whole scene (denoted by ‘Multi-shot’ in Table 2), while our approach can handle the whole scene with a single feed-forward operation (denoted by ‘Single-shot’ in Table 2).

mIoU vs. model size. We compare the accuracy of both Fast Point Transformer and MinkowskiNet with the different number of parameters. We build small network models by reducing the number of building blocks as MinkowskiNet does and maintaining the number of channels. Detailed illustration about network architecture is shown in the supplementary material. Table 4 shows the evaluation results.

These results imply that the proposed lightweight self-attention (LSA) layer can learn a 3D geometry more effectively than an over-parameterized sparse convolutional layer thanks to its dynamic kernel weights.

Table 6 shows the effects of attention types used in the proposed LSA layer. $\operatorname{cosine}(\cdot)$ handles the varying number of neighbors more effectively than $\operatorname{softmax}(\cdot)$ as shown in Table 6. However, as reported in local self-attention literature , additional usage of the similarity between query $\phi(\mathbf{g}_{i})$ and key $\xi(\mathbf{g}_{j})$ does not enhance the LSA layer.

5 3D Object Detection

We have conducted experiments on the ScanNet 3D object detection dataset, where a fine-grained point cloud representation is essential to detect and localize 3D objects.

Setups. For a fair comparison of Fast Point Transformer with previous methods , we use Torch-Points3D, an open-source library implemented by Chaton et al. for reproducible deep learning on 3D point clouds. Torch-Points3D sub-samples a fixed number of points from an input point cloud, which is widely used for PointNet++ to process a scene-level point cloud-like ScanNet. We notice that the library also sub-samples points for the voxel-based methods, such as MinkowskiNet , which is not a suitable experimental configuration. Therefore, we reproduce VoteNet with the MinkowskiNet backbone, which is denoted by MinkowskiNet† in Table 8, without input point sub-sampling, and we use the original experimental configurations. Additionally, we train a new VoteNet with the Fast Point Transformer backbone without any change of detection network (e.g., voting module).

Results. As shown in Table 8, the VoteNet model with Fast Point Transformer as a backbone outperforms other baselines with a large margin. The results show that the proposed continuous positional encodings that Fast Point Transformer uses can effectively encode point cloud representation and help the 3D detection task.

Conclusion

We have introduced the Fast Point Transformer and demonstrated its speed and accuracy on 3D semantic segmentation and 3D detection tasks. The experimental results on large-scale 3D datasets show that our approach is competitive to the best voxel-based method , and our network achieves 129 times faster inference time than the state-of-the-art, Point Transformer, with a reasonable accuracy trade-off in 3D semantic segmentation . However, there is room for improvement of the Fast Point Transformer at a small voxel size. In the future, we will explore architectures for Fast Point Transformer rather than U-shaped architectures that are initially designed for convolutional layers. Our code and data are going to be publicly available.

Acknowledgement. This work was supported by Qualcomm and the IITP grant (2021-0-02068: AI Innovation Hub and 2019-0-01906: AI Grad. School Prog.) funded by the Korea government (MSIT) and the NRF grant (NRF-2020R1C1C1015260).

Appendix A Appendix

In this appendix, we provide additional details and results of the proposed method, Fast Point Transformer.

In this section, we clarify the experimental settings for training models, latency evaluation, and model architectures in detail. Each experiment has been conducted with a fixed random seed for the reproducibility.

Latency evaluation. We describe the detailed setups that have been used during the inference time evaluation on Table 2 of the main paper. We measure the latency of each model with batch size 1 under the following environments:

CPU: Intel(R) Core(TM) i7-5930K CPU @ 3.50GHz

A.2 Analysis on Centroid-aware Voxelization

Color reconstruction. We conduct an experiment to evaluate the effeciveness of our centroid-aware voxelization. We compare ours and the conventional voxelization with the same setting from Table 5 of the main paper on ScanNet validation set. We reconstruct colors (RGB) of input point clouds with MinkowskiNet , optimized by $l$2-difference between input colors and reconstructed colors. As shown in Table A2, ours achieves a higher PSNR by 1.27 than the conventional one, showing the effectiveness of the centroid-aware property to mitigate quantization artifact.

A.3 Additional Experimental Results

In this section, we show further experimental results about the effect of model size on its performance, the proposed decomposition of positional encodings, and the class-wise IoU scores of both MinkowskiNet42† and our Fast Point Transformer on S3DIS Area 5 test dataset.

A.4 Time Complexity Analysis

In this section, we analyze the time complexity of neighbor search used in both voxel hashing-based methods including ours and point-based methods . We recap the reported time complexity as shown in Table A8.

MinkowskiNet and Fast Point Transformer require the same process for neighbor search since both methods benefit from voxel hashing. We analyze preparation and inference time complexity on Alg. 1 and Alg. 2, respectively. We denote ours as the representative method.

KPConv constructs a $k$-d tree before inference. With the official code of KPConv, we analyze both preparation and inference time in Alg. 3 and Alg. 4, respectively.

PointWeb uses a brute-force algorithm to search the $k$ nearest neighbors. We analyze the time complexity of the brute-force algorithm in Alg. 5.

PAConv and Point Transformer do not require preparation steps for neighbor search. Thus, we set the preparation time to constant time. For analyzing inference time, we have followed the official implementation. As both methods use the same algorithm for neighbor search, we denote PAConv as the representative method in Alg. 6.

A.5 Qualitative Results

In this section, we show further qualitative results of consistency scores, 3D semantic segmentation results, and 3D object detection on ScanNet . Figure A2 shows the point-wise consistency scores of MinkowskiNet42† and our Fast Point Transformer. In addition to this consistency, Fast Point Transformer predicts more accurate 3D semantic labels (Figure A3) and 3D bounding boxes (Figure A4) qualitatively.

A.6 Notations

References