NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video

Introduction

3D scene reconstruction is one of the central tasks in 3D computer vision with many applications. In augmented reality (AR) for example, to enable realistic and immersive interactions between AR effects and the surrounding physical scene, 3D reconstruction needs to be accurate, coherent and performed in real-time. While camera motion can be tracked accurately with state-of-the-art visual-inertial SLAM systems , real-time image-based dense reconstruction remains to be a challenging problem due to low reconstruction quality and high computation demands.

Most image-based real-time 3D reconstruction pipelines adopt the depth map fusion approach, which resemble RGB-D reconstruction methods like KinectFusion . Single-view depth maps from each key frame are first estimated with real-time multi-view depth estimation methods like . The estimated depth maps are later filtered with criteria like multi-view consistency and temporal smoothness, and fused into a Truncated Signed Distance Function (TSDF) volume. The reconstructed mesh can be extracted from the fused TSDF volume with the Marching Cubes algorithm . This depth-based pipeline has two major drawbacks. First, since single-view depth maps are estimated individually on each key frame, each depth estimation is from scratch instead of conditioned on the previous estimations even the view-overlapping is substantial. As a result, the scale-factor may vary even with the correct camera ego-motion. Due to depth inconsistencies between different views, the reconstruction result is prone to be either layered or scattered. One example is shown in the red boxes in Fig. 1, where the depth-based method struggles to produce coherent depth estimations on the chairs and wall. Second, since key-frame depth maps need to be estimated separately in overlapped local windows, geometry of the same 3D surface is estimated multiple times in different key frames, causing redundant computation.

In this paper, we propose a novel framework for real-time monocular reconstruction named NeuralRecon that jointly reconstructs and fuses the 3D geometry directly in the volumetric TSDF representation. Given a sequence of monocular images and their corresponding camera poses estimated by a SLAM system, NeuralRecon incrementally reconstructs local geometry in a view-independent 3D volume instead of view-dependent depth maps. Specifically, it unprojects the image features to form a 3D feature volume and then uses sparse convolutions to process the feature volume to output a sparse TSDF volume. With a coarse-to-fine design, the predicted TSDF is gradually refined at each level. By directly reconstructing the implicit surface (TSDF), the network is able to learn the local smoothness and global shape prior of natural 3D surfaces. Different from depth-based methods that predict depth maps for each key frame separately, the surface geometry within a local fragment window is jointly predicted in NeuralRecon, and thus locally coherent geometry estimation can be produced. To make the current-fragment reconstruction to be globally consistent with the previously reconstructed fragments, a learning-based TSDF fusion module using the Gated Recurrent Unit (GRU) is proposed. The GRU fusion makes the current-fragment reconstruction conditioned on the previously reconstructed global volume, yielding a joint reconstruction and fusion approach. As a result, the reconstructed mesh is dense, accurate and globally coherent in scale. Furthermore, predicting the volumetric representation also removes the redundant computation in depth-based methods, which allows us to use a larger 3D CNN while maintaining the real-time performance.

We validate our system on the ScanNet and 7-Scenes datasets. The experimental results show that NeuralRecon outperforms multiple state-of-the-art multi-view depth estimation methods and the volume-based reconstruction method Atlas by a large margin, while achieving a real-time performance at 33 key frames per second, ${\sim}10\times$ faster compared to Atlas. As shown in the supplementary video, our method is able to reconstruct large-scale 3D scenes from a video stream on a laptop GPU in real-time. To the best of our knowledge, this is the first learning-based system that is able to reconstruct dense and coherent 3D scene geometry in real-time.

Related Work

Multi-view Depth Estimation. The most related line of research is real-time methods for multi-view depth estimation. Before the age of deep learning, many renowned works in monocular 3D reconstruction have achieved good performance with plane-sweeping stereo and depth filters under the assumption of photo-consistency. optimize this line of research towards low power consumption on mobile platforms. Learning-based methods on real-time multi-view depth estimation try to alleviate the photo-consistency assumption with a data-driven approach. Notably, MVDepthNet and Neural RGB-$>$D use 2D CNNs to process the 2D depth cost volume constructed from multi-view image features. CNMNet further leverages the planar structure in indoor scenes to constrain the surface normals calculated from the predicted depth maps to obtain smooth depth estimation. These learning-based methods use 2D CNNs to process the depth cost volume to maintain a low computational cost for near real-time performance.

When the input images are high-resolution and offline computation is allowed, multi-view depth estimation is also known as the Multiple View Stereo (MVS) problem. PatchMatch-based methods have achieved impressive accuracy and are still the most popular methods applicable to high-resolution images. Learning-based approaches in MVS have recently dominated several benchmarks in terms of accuracy, but are only limited to processing mid-resolution images due to the GPU memory constraint. Different from the real-time methods, 3D cost volumes are constructed and 3D CNNs are used to process the cost volume as proposed in MVSNet . Some recent works improve this pipeline with a coarse-to-fine approach. Similar design can also be found in many learning-based SLAM systems .

All the above-mentioned works adopt single-view depth maps as intermediate representations. SurfaceNet takes a different approach and uses a unified volumetric representation to predict the volume occupancy. Recently, Atlas also proposes a volumetric design and direct predicts TSDF and semantic labels with 3D CNN. As an offline method, Atlas aggregates the image features of the entire sequence and then predicts the global TSDF volume only once with a decoder module. We further elaborate the relationship between the proposed method and Atlas in the supplementary material. The proposed method is also related to in terms of using recurrent networks for multi-view feature fusion. However, their recurrent fusion is applied to only the global features and their focus is to reconstruct single objects.

3D Surface Reconstruction. After depth maps are estimated and converted to point clouds, the remaining task for 3D reconstruction is to estimate the 3D surface position and produce the reconstructed mesh. In an offline MVS pipeline , Poisson reconstruction and Delaunay triagulation are often used to fulfill this purpose. Proposed by the seminal work KinectFusion , incremental volumetric TSDF fusion gets widely adopted in real-time reconstruction scenarios due to its simplicity and parallelization capability. improve KinectFusion by making it more scalable and robust. RoutedFusion changes the fusion operation from a simple linear addition into a data-dependent process.

Neural Implicit Representations. Recently, neural implicit representations have gained significant advances. Our work also learns a neural implicit representation by predicting SDF with the neural network from the encoded image features similar to PIFu . The key difference is that we are using sparse 3D convolution to predict a discrete TSDF volume, instead of querying the MLP network with image features and 3D coordinates.

Methods

To achieve real-time 3D reconstruction that is suitable for interactive applications, the reconstruction process needs to be incremental and the input images should be processed sequentially in local fragments . We seek to find a set of suitable key frames from the incoming image stream as input for the networks. To provide enough motion parallax while keeping multi-view co-visibility for reconstruction, the selected key frames should be neither too close nor far from each other. Following , a new incoming frame is selected as a key frame if its relative translation is greater than $t_{max}$ and the relative rotation angle is greater $R_{max}$. A window with $N$ key frames is defined as a local fragment. After key frames are selected, a cubic-shaped fragment bounding volume (FBV) that encloses all the key frame view-frustums is computed with a fixed max depth range $d_{max}$ in each view. Only the region within the FBV is considered during the reconstruction of each fragment.

2 Joint Fragment Reconstruction and Fusion

We propose to simultaneously reconstruct the TSDF volume of a local fragment $\mathbf{S}_{t}^{l}$ and fuse it with global TSDF volume $\mathbf{S}_{t}^{g}$ with a learning-based approach. The joint reconstruction and fusion is carried out in the local coordinate system. The definition of the local and global coordinate systems as well as the construction of FBV are illustrated in Fig. 1 of the supplementary material.

Image Feature Volume Construction. The $N$ images in the local fragment are first passed through the image backbone to extract the multi-level features. Similar to previous works on volumetric reconstruction , the extracted features are back-projected along each ray into the 3D feature volume. The image feature volume $\mathbf{F}_{t}^{l}$ is obtained by averaging the features from different views according to the visibility weight of each voxel. The visibility weight is defined as the number of views from which a voxel can be observed in the local fragment. A visualization of this unprojection process can be found in Fig.3 i.

Coarse-to-fine TSDF Reconstruction. We adopt a coarse-to-fine approach to gradually refine the predicted TSDF volume at each level. We use 3D sparse convolution to efficiently process the feature volume $\mathbf{F}_{t}^{l}$. The sparse volumetric representation also naturally integrates with the coarse-to-fine design. Specifically, each voxel in the TSDF volume $\mathbf{S}_{t}^{l}$ contains two values, the occupancy score $o$ and the SDF value $x$. At each level, both $o$ and $x$ are predicted by the MLP. The occupancy score represents the confidence of a voxel being within the TSDF truncation distance $\lambda$. The voxel whose occupancy score is lower than the sparsification threshold $\theta$ is defined as void space and will be sparsified. This representation of sparse TSDF volume is visually illustrated in Fig.3 iii. After the sparsification, $\mathbf{S}_{t}^{l}$ is upsampled by $2\times$ and concatenated with the $\mathbf{F}_{t}^{l+1}$ as the input for the GRU Fusion module (introduced later) in the next level.

Instead of estimating single-view depth maps for each key frame, NeuralRecon jointly reconstructs the implicit surface within the bounding volume of the local fragment window. This design guides the network to learn the natural surface prior directly from the training data. As a result, the reconstructed surface is locally smooth and coherent in scale. Notably, this design also leads to less redundant computation compared to depth-based methods since each area on the 3D surface is estimated only once during the fragment reconstruction.

GRU Fusion. To make the reconstruction consistent between fragments, we propose to make the current-fragment reconstruction to be conditioned on the reconstructions in previous fragments. We use a 3D convolutional variant of Gated Recurrent Unit (GRU) module for this purpose. As illustrated in Fig.3 ii, at each level the image feature volume $\mathbf{F}_{t}^{l}$ is first passed through the 3D sparse convolution layers to extract 3D geometric features $\mathbf{G}_{t}^{l}$. The hidden state $\mathbf{H}_{t-1}^{l}$ is extracted from the global hidden state $\mathbf{H}_{t-1}^{g}$ within the fragment bounding volume. GRU fuses $\mathbf{G}_{t}^{l}$ with hidden state $\mathbf{H}_{t-1}^{l}$ and produces the updated hidden state $\mathbf{H}_{t}^{l}$, which will be passed through the MLP layers to predict the TSDF volume $\mathbf{S}_{t}^{l}$ at this level. The hidden state $\mathbf{H}_{t}^{l}$ will also be updated to global hidden state $\mathbf{H}_{t}^{g}$ by directly replacing the corresponding voxels. Formally, denoting $z_{t}$ as the update gate, $r_{t}$ as the reset gate, $\sigma$ as the sigmoid function and $W_{*}$ as the weight for sparse convolution, GRU fuses $\mathbf{G}_{t}^{l}$ with hidden state $\mathbf{H}_{t-1}^{l}$ with the following operations:

Intuitively, in the context of joint reconstruction and fusion of TSDF, the update gate $z_{t}$ and forget gate $r_{t}$ in the GRU determine how much information from the previous reconstructions (i.e. hidden state $\mathbf{H}^{l}_{t-1}$) is fused to the current-fragment geometric feature $\mathbf{G}_{t}^{l}$, as well as how much information from the current-fragment will be fused into the hidden state $\mathbf{H}^{l}_{t}$. As a data-driven approach, the GRU serves as a selective attention mechanism that replaces the linear running-average operation in conventional TSDF fusion . By predicting $\mathbf{S}_{t}^{l}$ after the GRU, the MLP network can leverage the context information accumulated from history fragments to produce consistent surface geometry across local fragments. This is also conceptually analogous to the depth filter in a non-learning-based 3D reconstruction pipeline , where the current observation and the temporally-fused depths are fused with the Bayesian filter. The effectiveness of joint reconstruction and fusion is validated in the ablation study.

Integration to the Global TSDF Volume. At the last coarse-to-fine level, $\mathbf{S}_{t}^{3}$ is predicted and further sparsified to $\mathbf{S}_{t}^{l}$. Since the fusion between $\mathbf{S}_{t}^{l}$ and $\mathbf{S}_{t}^{g}$ has been done in GRU Fusion, $\mathbf{S}_{t}^{l}$ is integrated into $\mathbf{S}_{t}^{g}$ by directly replacing the corresponding voxels after being transformed into the global coordinate. At each time step $t$, Marching Cubes is performed on $\mathbf{S}_{t}^{g}$ to reconstruct the mesh.

3 Implementation Details

We use torchsparse as the implementation of 3D sparse convolution. The image backbone is a variant of MnasNet and is initialized with the weights pretrained from ImageNet. Feature Pyramid Network is used in the backbone to extract more representative multi-level features. The entire network is trained end-to-end with randomly initialized weights except for the image backbone. The occupancy score $o$ is predicted with a Sigmoid layer. The voxel size of the last level is $4cm$ and the TSDF truncation distance $\lambda$ is set to $12cm$. $d_{max}$ is set to $3m$. $R_{max}$ and $t_{max}$ are set to 15°and $0.1m$ respectively. $\theta$ is set to $0.5$. Nearest-neighbor interpolation is used in the upsampling between coarse-to-fine levels.

Experiments

In this section, we conduct a series of experiments to evaluate the reconstruction quality and different design considerations of NeuralRecon.

Datasets. We perform the experiments on two indoor datasets, ScanNet (V2) and 7-Scenes . The ScanNet dataset contains 1613 indoor scenes with ground-truth camera poses, surface reconstructions, and semantic segmentation labels. There are two training/validation splits commonly used in previous works (defined in and ) for the ScanNet dataset. We use the same training and validation data with the corresponding baseline methods to make a fair comparison. The 7-Scenes dataset is another challenging RGB-D dataset captured in indoor scenes. Following the baseline method , we use the model trained on ScanNet to perform the validation on 7-Scenes.

Metrics. The 3D reconstruction quality is evaluated using 3D geometry metrics presented in , as well as standard 2D depth metrics defined in . The definitions of these metrics are detailed in the supplementary material. Among these 3D and 2D metrics, we consider F-score as the most suitable metrics to measure 3D reconstruction quality since both the accuracy and completeness of the reconstruction are considered.

Baselines. We compare our method with the following baseline methods in three categories: 1) Real-time methods for multi-view depth estimation . Due to the efficiency constraints, the estimated depth accuracy by these methods is rather limited. We compare with these methods to demonstrate the better reconstruction accuracy of NeuralRecon given the same efficiency. 2) Multiple View Stereo methods . These offline methods have much higher accuracy compared to real-time methods. These baselines are used to demonstrate that NeuralRecon achieves a reconstruction quality on-par with offline methods but runs in real-time. 3) Learning-based SLAM methods . These monocular SLAM methods estimate camera poses and perform reconstruction simultaneously, thus the scale factor of pose and depth is usually not accurately estimated. For a fair comparison, we use ground-truth camera poses for these methods and apply a scaling factor to the predicted depth map using ground-truth depth. Among all these baseline methods, GPMVS and Atlas are the most relevant real-time and offline methods, respectively.

Evaluation Protocols. Since our method does not estimate depth maps explicitly, we render the reconstructed mesh to the image plane and obtain depth map estimations . Key frames used for evaluation are sampled from the video sequence with an interval of 10 frames for both depth-based methods and Atlas. Following , are fine-tuned on ScanNet. To evaluate depth-based methods in 3D, we use the point cloud fusion to obtain the 3D reconstruction following Atlas. For other depth-based methods, we use the standard TSDF fusion proposed in . For the reasons we detailed in the supplementary material, in order to make a fair comparison with Atlas, we also report the evaluation results using the double-layered mesh (same as Atlas). The evaluation of 3D geometry on 7-Scenes uses the single-layered mesh. We also evaluate the depth filtering operation with multi-view consistency check, which will be elaborated in the supplementary material.

2 Evaluation Results

ScanNet. 2D depth metrics and 3D geometry metrics are used on the ScanNet dataset. The 3D geometry evaluation results are shown in Tab. 1. Our method produces much better performance than recent learning-based methods and achieves slightly better results than COLMAP. We believe that the improvements come from the joint reconstruction and fusion design achieved by the GRU Fusion module. Compared to depth-based methods, NeuralRecon can produce coherent reconstructions both locally and globally. Our method also surpasses the volumetric baseline method Atlas on the accuracy, precision, and F-score. The improvements potentially come from the design of local fragment separation in our method, which can act as a view-selection mechanism that avoids irrelevant image features to be fused into the 3D volume. In terms of completeness and recall, the proposed method has an inferior performance compared to both depth-based methods and Atlas. Since depth-based methods predict pixel-wise depth maps on each view, the coverage of their predictions is high by nature, but with the cost of accuracy. Being an offline approach, Atlas has the advantage of having a global context from the entire sequence before predicting the geometry. As a result, Atlas sometimes achieves even better completeness compared to the ground-truth due to its TSDF completion capability. However, Atlas tends to predict over-smoothed geometries, and the completed regions may be inaccurate. As for 2D depth metrics, NeuralRecon also outperforms previous state-of-the-art methods for almost all 2D depth metrics, as shown in Tab. 2.

7-Scenes. 2D depth metrics and 3D geometry metrics are evaluated on the 7-Scenes dataset. As shown in Tab. 3, our method achieves comparable performance to the state-of-the-art method CNMNet and outperforms all other methods. We believe that the accuracy of the proposed method can be further improved by leveraging the planar structure information as in CNMNet. Since the model used here is only trained on ScanNet, the results also demonstrate that NeuralRecon can generalize well beyond the domain of the training data.

Efficiency. We also report the average running time of the baselines and our method in Tab. 1. Only the inference time on key frames is computed. A detailed timing analysis for each module of NeuralRecon is presented in Table 4. For volumetric methods (Atlas and ours), the running time is obtained by dividing the time of reconstructing the TSDF volume of a local fragment by the number of key frames in the local fragment. Notice that the time for TSDF fusion is not included for depth-based methods. The running time for and NeuralRecon is measured on an NVIDIA RTX 2080Ti GPU. We use running time reported in and for and , respectively.

As shown in Tab. 1, our time cost is 30ms per key frame, achieving real-time speed at 33 key frames per second and outperforming all previous methods. Specifically, our method runs $\sim$10$\times$ faster than Atlas, and 77$\times$ faster than Consistent Depth. Predicting the volumetric representation removes the redundant computation in depth-based methods, which contributes to the fast running speed of our method. Compared to Atlas, incrementally reconstructing geometry in local fragment avoids processing a huge 3D volume, leading to a faster speed than Atlas. The use of sparse convolution also contributes to the superior efficiency of NeuralRecon.

3 Ablation Study

In this section, we conduct several ablation experiments on the ScanNet dataset to discuss the effectiveness of components in our method.

GRU Fusion. We validate the GRU Fusion design by comparing rows from (i) to (iv) in Tab. 5.

To validate the benefit of feature fusion, we compare row (i) and row (ii) in Tab. 5. Using feature fusion with the average operation obtains nearly 5% improvement for the precision metric than conventional linear TSDF fusion. Visualization in Fig. 5 shows that feature fusion with the average operation can reconstruct smoother geometry. These results demonstrate that feature fusion can be more effective than TSDF fusion using the same average operation.

Comparing row (ii) and row (iii) in Tab. 5 shows that replacing average operation with GRU gives 4% improvement in terms of recall. The mesh in Fig. 5 (iii) is also more complete than that in Fig. 5 (ii). These results demonstrate that the GRU is more effective to selectively integrate only the consistent information from the current-fragment to the hidden state.

The recalls in row (iii) and row (iv) in Tab. 5 show that fusion in the fragment bounding volume can produce much more complete results. Visualization results in Fig. 5 (iii) and (iv) show that, with fusion in the fragment bounding volume, our method produces fewer artifacts on the ground. Fusion in the fragment bounding volume can leverage the context information in boundaries and produce more consistent and complete surface estimation.

Number of views. We set 5, 7, 9 and 11 views as the length of a fragment respectively. As shown in row (v) in Tab. 5, the F-score has over 2% improvement when 9 views are used as a fragment. As shown in visualization results in Fig. 5 (v), with more views in a fragment, the geometry can be reconstructed more accurately compared to Fig. 5 (iv).

Qualitative Results. We provide the qualitative results and the corresponding analysis in Fig. 4.

Conclusion

In this paper, we introduced a novel system NeuralRecon for real-time 3D reconstruction with monocular video. The key idea is to jointly reconstruct and fuse sparse TSDF volumes for each video fragment incrementally by 3D sparse convolutions and GRU. This design enables NeuralRecon to output accurate and coherent reconstruction in real-time. Experiments show that NeuralRecon outperforms state-of-the-art methods in both reconstruction quality and running speed. The sparse TSDF volume reconstructed by NeuralRecon can be directly used in downstream tasks like 3D object detection, 3D semantic segmentation and neural rendering. We believe that, by jointly training with the downstream tasks end-to-end, NeuralRecon enables new possibilities in learning-based multi-view perception and recognition systems.

Acknowledgement. The authors would like to acknowledge the support from the National Key Research and Development Program of China (No. 2020AAA0108901), NSFC (No. 61806176), and ZJU-SenseTime Joint Lab of 3D Vision.

References