Real-time Progressive 3D Semantic Segmentation for Indoor Scene

Introduction

Recent hardware advances in consumer-grade depth cameras have made high-quality reconstruction of indoor scenes feasible. RGB-D images have been used to boost the robustness of numerous scene understanding tasks in computer vision, such as object recognition, object detection, and semantic segmentation. While scene understanding using color or RGB-D images is a well explored topic , good solutions for the same task in the 3D domain have been highly sought after, particularly, those can produce accurate and high-quality semantic segmentation.

In this work, we propose a (near) real-time method for high-quality dense semantic segmentation of 3D indoor scene. The backbone of our work is a higher-order conditional random field (CRF) designed to infer optimal segmentation labels from the predictions of a deep neural network. The CRF runs in tandem with a revised pipeline for real-time 3D reconstruction using RGB-D images as input. In contrast to traditional dense model, our CRF accepts additional higher-order constraints from unsupervised object analysis, resulting in high-quality segmentation. An example output from our proposed method is shown in Figure 1. Experiments proved that our method is capable of producing high-quality semantic segmentation and achieve adequate temporal consistency. In summary, our contributions are:

A higher-order conditional random field that can resolve noisy predictions from a deep neural network into a coherent 3D dense segmentation, using additional object-level information.

An extended reconstruction pipeline, including an efficient voxel clustering technique, for efficient (near) real-time full-scene inference while scanning.

A thorough evaluation of state-of-the-art real-time semantic segmentation algorithms on two large scale indoor datasets, namely SceneNN and ScanNet .

Beyond category-based semantic segmentation, we also extend our method to instance-based semantic segmentation, and provide the first evaluation of real-time instance segmentation on SceneNN dataset.

Related Work

In their seminal work, Silberman et al. proposed a technique to segment cluttered indoor scenes into floor, walls, objects and their support relationships. Their well-known NYUv2 dataset has since sparked new research interests in semantic segmentation using RGB-D images. Long et al. adapted neural networks originally trained for classification to solve semantic segmentation by appending a fully connected layer to the existing architecture. This method, however, tends to produce inaccuracies along object boundaries. Since then, different techniques has been proposed to address this issue. Some recent works also explored instance segmentation , but such techniques only work in 2D.

In the 3D domain, a few datasets for 3D scene segmentation have also been proposed . Early techniques focused on solving the problem by exploiting 3D volumes. For example, Song et al. and Dai et al. proposed a network architecture for semantic scene segmentation and completion at the same time. Point-based deep learning took another direction and attempted to learn point representation for segmentation directly from unordered point clouds. While the results from these neural networks are impressive, they only take as input a small point cloud of a few thousand points. To address large-scale or structural point cloud, clustering techniques such as super-points or hierarchical data structures such as octree and kd-tree have been proposed. Hybrid methods such as SEGCloud turns the point clouds into volumes for prediction with a neural network and then propagates the results back to the original point cloud.

Instead of directly processing in 3D, multiple view techniques focused on transferring 2D segmentation to 3D. Other methods further exploit object cues such as spatial context . Our method is based on multi-view segmentation as such techniques scale better to large-scale scenes. Concurrently, we also aim to achieve real-time performance with progressive scene reconstruction. We would focus our discussion to the most relevant interactive and real-time techniques.

Real-time semantic segmentation.

Our real-time semantic segmentation system requires an online dense 3D reconstruction system. KinectFusion showed us how to construct such system. To overcome the spatial constraints in the original KinectFusion implementation, which prohibits large-scale 3D scanning, Nießner et al. used voxel hashing to reduce the memory footprint. Valentin et al. proposed an interactive scanning system where the segmentation is learnt from user inputs. Unlike them, our method is completely automatic without the need of user interaction, and thus more suitable for robotics applications. Our method is based on a segmentation prediction with 2D deep neural networks, a 2D-3D label transfer and optimization with a conditional random field (CRF). To our knowledge, the closest works to ours in this aspect is from the robotics community . Early methods utilized random forest classifiers to initialize the CRF but their end-to-end pipeline performance was far from real time. Similar to our approach, McCormac et al. utilized segmentation predictions from a deep neural network and achieved real-time performance on sparse point cloud. In comparison, our method preserves surface information completely by working with an on-the-fly sparse volume representation from Voxel Hashing , and introduce a higher-order conditional random field model to refine 3D segmentation.

Conditional random field.

The CRF model, often containing unary and pairwise terms, is commonly used as post-processing step to address noise in semantic segmentation. Krähenbühl and Koltun proposed an efficient message passing method to perform inference on a fully-connected model. Recently, with the immense advances in deep learning, it is possible to embed CRF into neural networks and its parameters can be learnt jointly with the network via back-propagation. While representing CRF by a recurrent neural network is advantageous, applying such end-to-end framework to our problem poses some challenges. First in the context of progressive 3D reconstruction and segmentation, 2D predictions from multiple views have to be combined to produce the labeling of 3D model, which is not supported in the previous method where only the segmentation of one single image is predicted. Second, their methods is computationally demanding which does not fit our real-time requirement. Third, the number of 2D images used to calculate the unaries is not fixed, compared to using only one input image as in previous approaches. In this work, we instead run the CRF separately on 3D after processing 2D semantic predictions from a convolutional neural network.

CRF is also extended with high-order potentials to further improve coherency in the label prediction. For example, Zhu et al. explored high-order CRF for co-segmentation on images. Yang et al. uses a hierarchical CRF with potentials from super-pixels on images for fast outdoor scene segmentation. The CRF model we propose in this work is a higher-order CRF that includes object cues for indoor scenes and works in tandem with the geometry reconstruction. Our idea is that to obtain a coherent, high-quality segmentation, vertices in the same object should be consider as a whole in the model. Moreover, noises and inconsistencies should be fixed regularly as the user scans through the scene.

Real-time RGB-D Reconstruction

We now introduce our proposed method for the progressive dense semantic segmentation problem. An overview of our framework is shown in Figure 2.

Our online scanning system is built on top of the Voxel Hashing pipeline, reconstructing both geometric and semantic information of the scene in real time. In principle, given an incoming frame prediction from CNN, we must update the semantic label for each active voxel accordingly, using the same integration process as described in KinectFusion . For this problem, McCormac et al. store a full discrete probability distribution in each voxel, and update it by using recursive Bayesian rule. However, doing so requires a large amount of memory and does not scale well with large number of semantic classes. We employ the update process proposed by Cavallari and Di Stefano , where each voxel only stores the current best label and its confidence.

2 Progressive super-voxel clustering

Now we explain in details our super-voxel clustering method, which will provide a new domain to define our CRF with higher-order constraints. Our super-voxel clustering method resembles previous local k-means clustering techniques such as VCCS or SLIC . The main difference in our super-voxel clustering method is that, to amortize the computation cost, we create super-voxels in a progressive manner, performing one clustering iteration at a time, which will adapt better to the changes in the current reconstructed scene. In our system, we consider common features such as voxel color and position to define the distance measure $D$:

where $D_{c}$ and $D_{s}$ are the color and spatial distances, with $n_{c}$ and $n_{s}$ act as the normalizers; $\alpha$ and $\beta$ control the relative weighting of color and spatial distances. In all of our experiments, we set $\alpha$ and $\beta$ to 1; the normalization values $n_{c}$ and $n_{s}$ are based on the chosen voxel size which is $0.008m$ and the CIELab color space. Here one can further utilize voxel normals for the distance measure but we found that the quality of the clustering does not improve much despite of the expensive cost to compute normals per voxel. Another possible extension is to consider features provided by the 2D semantic segmentation network in the distance measure. However, the memory storage per voxel would be very costly because each feature vector often has at least tens of floating point numbers. Some compressions might help in this case.

Suppose that an existing set of super-voxels are already provided. For an incoming RGB-D frame at time $t$, after camera pose estimation, we can find out the current active set $V_{t}$ of voxels using an inside/outside check on the current camera frustum. Our goal is to assign each of these voxels into a super-voxel (or cluster). This process is as follows: first new seeds are sampled on uninitialized regions, based on a chosen spatial interval $S$. For each active voxel, we assign it to the nearest cluster according to the distance in Equation 1. Next, we update the centers information based on the new cluster assignment. This process is repeated for every incoming RGB-D frame, providing a “live” unsupervised over-segmentation of the scene.

Our progressive super-voxel building scheme fits well into the common dense RGB-D reconstruction pipelines such as KinectFusion or Voxel Hashing , and can be implemented efficiently on the GPU. In practice, we only consider voxels close to the surface, based on their distance-to-surface values. Performing inference on these super-voxels significantly reduces the domain size of our CRF, and thus paves the way for real-time semantic segmentation.

3 Real-time object proposal

For 3D object proposal, Karpathy et al. presented a method for discovering object models from 3D meshes of indoor environments. Their method first generates object candidates by over-segmenting the scene on different thresholds. The candidates are then evaluated and suppressed based on geometric metrics to produce the final proposals. Kanezaki proposed an extension of selective search for object proposal on 3D point cloud.

One common drawback of these methods is their high computation cost, since they require a costly object analysis on different scales. This process has to be done for every update, which hinders real-time performance. In this work, we explore on a new direction for object proposal, in which we propose object based on statistical evidences.

Our object proposal is come from a simple observation: given an object and multiple observations, it should be identified as an object in most of the corresponding 2D semantic predictions. Hence, for each incoming RGB-D frame, we update the objectness score of a voxel given its current predicted label. Specifically, we decrease the objectness score if the prediction is a non-object label, i.e. wall, floor, or ceiling; and increase it otherwise. To perform object proposal, we employ an efficient graph-based segmentation algorithm from Felzenszwalb and Huttenlocher . The edge weight between two super-voxels $i$ and $j$ is defined as $w_{i,j}=w^{\alpha}_{i,j}+w^{\eta}_{i,j}+w^{\omega}_{i,j}$ where $w^{\alpha}_{i,j}$, $w^{\eta}_{i,j}$, and $w^{\omega}_{i,j}$ are the edge weight for voxel color, normal, and objectness, respectively. We normalize the each of the weights accordingly. To reduce computation cost, we only compute the terms using representative values from super-voxel centroids.

Higher-order CRF Refinement

Using CRF as a post-processing step is a common technique in semantic segmentation. However, for real-time applications, there are two limitations that we must address. First is the classification errors caused by inconsistencies, sometimes known as “bleeding”, that is also reported by Valentin et al. . The second issue is scalability, since the number of vertices in the graph grows to millions during scanning, causing CRF optimizations to become much slower over time. In this work, we address both limitations by introducing a CRF model with higher-order constraints on super-voxels to perform online segmentation. This model is lightweight and very easy to compute, allowing it to work on a wide range of indoor scenes, while remaining computationally efficient for real-time use.

Given the above definitions, we define a graph $\mathcal{G}$ where each vertex is from $\mathbf{X}$. In addition, let $\mathcal{C}$ be the set of cliques in $\mathcal{G}$, given by an object proposal method. For every clique $r\in\mathcal{C}$, we can select a corresponding set of random variables $\mathbf{x}_{r}$ that belongs to $r$. Our CRF model introduces three new types of higher-order potential, namely objectness potential $\psi^{O}$, consistency potential $\psi^{C}$ and object relationship potential $\psi^{R}$. These terms are later explained in Section 4.1, 4.2, and 4.3, respectively. Our complete CRF model is then defined as

where $\varphi(x_{i})$ and $\psi^{P}(x_{i},x_{j})$ are the unary and pairwise terms used in the traditional dense CRF model. The unary term represent the prediction from a local classifier. In our case, it is obtained from fusing CNN predictions during reconstruction.

The pairwise (smoothness) potential $\psi^{P}(x_{i},x_{j})$ is parameterized by a Gaussian kernel

where $\mu_{ij}$ is the label compatibility function between $x_{i}$ and $x_{j}$ given by the Potts model; $p_{i}$ and $n_{i}$ are the location and normal of the $i^{th}$ super-voxel; $\theta_{\alpha}$ and $\theta_{\beta}$ are standard deviations of the kernel.

The term $\psi^{O}(\mathbf{x}_{r})$ captures the mutual agreement between the objectness score of a clique and its semantic label. Ideally, we would want a clique with low objectness score to take a non-object label, i.e. wall, floor, or ceiling; and inversely. To model the objectness potential of a clique, we first introduce latent binary random variables $y_{1},\dots,y_{\mid\mathcal{C}\mid}$. $y_{k}$ can be interpreted as follows: if the $k^{th}$ proposal has been found to be an object, then $y_{k}$ is 1, otherwise it will be 0. Let $\mathcal{O}$ be the subset of $\mathcal{L}$, which comprises of object classes in the label space. We can then define our objectness potential

where $[\cdot]$ is a function that converts a logical proposition into $1$ if the condition is satisfied, otherwise it would be . The purpose of this term is to correct misclassification errors in the local classifier, based on external unsupervised information from object proposal.

2 Label consistency

The term $\psi^{C}(\mathbf{x}_{r})$ enforces regional consistency in semantic segmentation. Since we want vertex labels in the same clique to be homogeneous, the cost function penalizes label based on its frequency in the clique. Let $f_{r}(l_{k})$ be the normalized frequency of label $l_{k}\in\mathcal{L}$ inside the $r^{th}$ clique, which is of the range between and $1$. The consistency cost will be the entropy of the underlying distribution:

This term dampens infrequent labels in a clique. In experiments, We observed that the label consistency cost helps fixing low frequency errors in the output segmentation.

3 Region relationship

The relationship potential $\psi^{R}$ encodes the relation between two regions (cliques) and their semantic labels. This cost is applied on neighboring regions, based on super-voxel connectivity. In our model, the term $\psi^{R}(\mathbf{x}_{r},\mathbf{x}_{q})$ is defined based on the co-occurrence of class labels in the regions. Specifically, let $\mathcal{E}(\mathcal{C})\subset\mathcal{C}\times\mathcal{C}$ be the edges between connected cliques. The object relationship cost between $\mathbf{x}_{r}$ and $\mathbf{x}_{q}$ is defined as follows,

where $\Lambda_{l_{i},l_{j}}$ is the co-occurrence cost based on the class labels $l_{i}$ and $l_{j}$ and designed such that the more often $l_{i}$ and $l_{j}$ co-occur, the greater $\Lambda_{l_{i},l_{j}}$ is. This cost acts like a prior to prevent uncommon label transition, e.g. chair to ceiling, ceiling to floor, etc; and can be learnt beforehand. $f_{r}$ and $f_{q}$ are the label frequencies, as presented in (5).

In our CRF model, each term is accompanied with a weight to balance their values that we omit them in our formulas for better clarity. We learn these weights by grid search, and keep them unchanged in all of the experiments.

Finally, semantic segmentation can be done by minimizing the energy function $E(\mathbf{X})$ defined in (2). In this paper, we adopt the variational mean field method for efficiently optimizing $E(\mathbf{X})$. Details of the inference process can be found in the supplementary material.

4 Temporal consistency

We support temporal consistency with a simple modification of the unary term as follows. To minimize storage, let us only consider time $t-1$ and time $t$. The unary term becomes a weighted sum that takes as input the final labels at time $t-1$ ($\mathbf{X}_{CRF}$, after CRF of time $t-1$) and the CNN predicted labels at the time $t$ ($\mathbf{X}_{predicted}$, before CRF): $\mathbf{X}_{unary}^{t}=\tau\mathbf{X}_{predicted}^{t}+(1-\tau)\mathbf{X}_{CRF}^{t-1}$ where $\mathbf{X}$ are the label probabilities, and $\tau\in$ is a scalar value. Smaller $\tau$ favors temporal consistency. We set $\tau$ empirically by plotting the segmentation accuracy with multiple $\tau$. Our experiment (see supplementary) shows that $\tau=0.5$ strikes a balance between accuracy and temporal consistency.

5 Instance segmentation

Beyond category-based semantic segmentation, we extend our technique to support instance-based semantic segmentation in real time, which we refer to as instance segmentation for brevity. The key change is that CRF model now outputs instance IDs instead of class segmentation labels. Other terms and the optimization process are kept unchanged.

A straightforward approach for instance segmentation would be utilizing a deep neural network that can perform instance-based segmentation in 2D, and then propagate the predictions from 2D to 3D as in the category-based semantic segmentation case. However, this approach requires us to track the instance IDs over time, which is in fact a challenging problem, since the networks, e.g., , can only predict one frame at a time.

Our solution is to combine category-based semantic segmentation network with the following instance-based segmentation to yield instance IDs. For each vertex $x_{i}$ in the CRF, we have to define probabilities over every possible instance IDs. The label space, $\mathcal{L}=\{l_{1},l_{2},\dots,l_{L}\}$, would be the set of all instance IDs in the current 3D reconstruction. Performing CRF inference on the entire set of instance IDs would be infeasible. Here we reduce the problem size by first filtering out the instance IDs that are not in the current camera frustum at time $t$, giving a reduced label space $\mathcal{L}^{t}$. Our higher-order CRF will only optimize instance labels of super-voxels in the camera frustum, instead of the entire scene as before. The result is then fused into the current model.

Another issue in progressive instance segmentation is how to update the label space $\mathcal{L}$, since online scanning will continuously introduce new instances to our model. We tackle this problem by creating a special unknown instance ID. All of the newly scanned voxels will be initialized with unknown. After each CRF inference step, the largest connected component, which is based on category, belongs to the unknown instance will be spawned as a new instance. We also update the set of instance IDs accordingly.

Experiments

Runtime analysis is performed on a desktop with an Intel Core i7-5820K 3.30GHz CPU, 32GB RAM, and an NVIDIA Titan X GPU. The average runtime breakdown of each step in the pipeline is demonstrated in Figure 4. Specifically, it takes $309.3$ms on average to run a single forward pass of neural network. Building super-voxels takes $34.1$ms. CRF with higher-order constraints requires additional $57.9$ ms. As can be seen, over time when more regions in the scene are reconstructed, our semantic segmentation still takes constant running time on average.

We compared our online approach to the reference offline approach that runs CNN prediction every frame (Table 5). We see that the accuracy of our online method (Table 5) is about 5% lower on average, but the speed gain is more than 8 times. Our system runs at 10-15Hz. With the same CNN predictions, direct fusion method runs at 17-20Hz, and SemanticFusion runs at 14-16Hz. Note that such methods do not constrain label consistency.

Since our method can be run in real time, we evaluate the segmentation accuracy over time. For every scene, we measure the accuracy every $100$ frames. The progressive segmentation results are shown in Figure 6.

The results suggest that our method consistently outperforms other methods in a long run, not just only at a certain time period. In addition, we observe that the accuracy over time sometimes still fluctuates slightly due to the lack of full temporal constraints among the CNN predictions. Addressing this issue could be an interesting future work.

To further understand the performance of our CRF model, we carry out an ablation study to evaluate the effects of each CRF term on the result segmentation. We execute three runs on $10$ scenes, each run enables only one term in our CRF model, and record their performances. Figure 5 visualizes the results on these $10$ scenes. In general, running full higher-order model achieves the best performance. Enabling individual term is able to outperform the base dense CRF model. The consistency term contributes the most in the performance boost, which validates our initial hypothesis that object-level information is crucial when performing dense semantic segmentation.

To evaluate our instance segmentation results, We use the average precision metric with minimal 50% overlap. The results are shown in Table 5. Figure 3 visualizes the instance segmentation in two indoor scenes using our approach. Such results could serve as a baseline to compare with more sophisticated real-time 3D instance segmentation technique in the future.

Our proposed system demonstrates the capability to integrate semantic segmentation into real-time indoor scanning by optimizing the predictions from a 2D neural network with a novel higher-order CRF model. The results and ground truth category-based and instance-based semantic segmentation will be made publicly available. The results from our system can further be used in other interactive or real-time applications, e.g., furniture arrangement , or object manipulation and picking in robotics.

This research project is partially supported by an internal grant from HKUST (R9429).