Learning Depth with Convolutional Spatial Propagation Network

Introduction

Depth estimation from images, i.e., predicting absolute per-pixel distance to the camera has many applications in practice, such as augmented reality (AR), autonomous driving , robotics . It also serves as a foundation to support other computer vision problems, such as 3D reconstruction and recognition .

As illustrated in Fig. 1, in this paper, we are particularly interested in two depth estimation tasks that are practical in real applications: depth completion and stereo matching.

Depth completion (Fig. 1(a)), a.k.a. sparse to dense depth conversion is a task of converting sparse depth samples to a dense depth map given the corresponding image . It can be widely applied in robotics and autonomous driving, where depth perception is often acquired through LiDAR , that typically generates sparse but accurate depth measurement. By combining the sparse measurements with a reference image, we could generate a full-frame dense depth map.

For this task, recent efforts have yielded high-quality outputs by taking advantage of deep fully convolutional neural networks and a large amount of training data from indoor and outdoor environments . The improvement lies mostly in more accurate estimation of global scene layout and scales with advanced networks, such as VGG or ResNet , and better local structure recovery through deconvolution operations , skip-connections or up-projection . Nevertheless, upon a closer inspection of the outputs from a contemporary approach for depth completion (Fig. 1(a) 3rd column), the predicted depth map is still blurry and often do not align well with the given image structure such as object silhouette. This motivates us to adopt linear propagation models for tackling the issue.

However, directly refining with bilateral filtering using pixel RGB (Fig. 1(a) 4th column) omits global image context, which produces noisy depth estimation. Most recently, Liu et al. propose to learn the image-dependent affinity through a deep CNN with spatial propagation networks (SPN), yielding better results comparing to the manually designed affinity on image segmentation. However, its propagation is performed in a scan-line or scan-column fashion, which is serial in nature. For instance, when propagating left-to-right, pixels at right-most column must wait the information from the left-most column to update its value.

Intuitively, depth refinement should not be order dependent. As such, we propose Convolutional Spatial Propagation Networks (CSPN), where the depth values at all pixels are updated simultaneously within a local convolutional context. The long range context is obtained through a recurrent operation. Fig. 1(a) shows an example, the depth estimated from CSPN (6th column) is more accurate than that from SPN (5th column). In our experiments, our parallel update scheme leads to significant performance improvement in both speed and quality over the serial ones such as SPN.

More specifically, in this paper, we consider three important requirements for a depth completion algorithm: (1) The dense depth map recovered should align with image structures; (2) The depth value from the sparse samples should be preserved, since they are usually from a reliable sensor; and (3) The transition between sparse depth samples and their neighboring depths should be smooth and unnoticeable. In order to satisfy those requirements, we first add mirror connections based on the network from to learn better image structural features. Then, we embed the sparse depth samples into the propagation process in order to keep the depth value at sparse points. Finally, our CSPN recovers image structures in detail and satisfies the transition smoothness requirements. In our experiments, we show our recovered depth map with just 500 depth samples produces much more accurately estimated scene layouts and scales than other SOTA strategies.

Stereo Matching (Fig. 1(b)), where a pair of calibrated images, commonly from a stereo camera , are used as inputs. Specifically, for pixel $(x,y)$ in the reference image (1st column), if its corresponding disparity is $d_{x,y}$, then the depth of this pixel could be calculated by $\frac{f*B}{d_{(x,y)}}$, where $f$ is the camera’s focal length and $B$ is the distance between two camera centers.

Current SOTA methods for stereo also rely on the advancement of deep networks . Most recently, GCNet learns to incorporate geometrical context directly from the data, employing 3D Convolutions (3DConv) over the disparity cost volume. PSMNet adopts a similar idea, while induces extensions at scale space using spatial feature pooling at the end of feature encoder and multi-scale outputs from their stacked hourglass networks with 3DConv. This motivates us to lift the 2D spatially propagating CSPN to 3D, where information can also propagate within the disparity space and scale space, yielding more accurate estimated results because of better details and less noises (as shown in Fig. 1(b) 4th column).

Last but not least, we regard the commonly used spatial pyramid pooling (SPP) and atrous SPP (ASPP) for multi-scale feature integration as special cases of CSPN. With such a perspective, we propose to learn an affinity matrix for feature aggregation within each pyramid and across pyramids, yielding further performance boost in both tasks than their original variants.

We perform various experiments to validate our approaches over several popular benchmarks for depth estimation. For depth completion, NYU v2 and KITTI are adopted. In both datasets, our approach is significantly better (about $30\%$ relative improvements in most key measurements) than previous deep learning based state-of-the-art algorithms . More importantly, it runs efficiently, yielding up to 5$\times$ acceleration over SPN on large images. For stereo depth estimation, the Scene Flow and KITTI Stereo datasets are adopted, and we rank the $1_{st}$ on both the KITTI Stereo 2012 and 2015 benchmarks http://www.cvlibs.net/datasets/kitti/eval_stereo_flow.php?benchmark=stereo.

In summary, this paper has the following contributions:

We propose convolutional spatial propagation networks (CSPN) that learn the affinity directly from images. It has shown to be more efficient and accurate for depth estimation than the previous SOTA propagation strategy , without sacrificing the stability of linear propagation.

We extend CSPN to the task of depth completion by embedding the provided sparse depth samples into the propagation process, which also guarantees that the sparse input depth values are preserved in the final depth map.

We lift CPSN to 3D CSPN for stereo depth estimation, which explores the correlation within both discrete disparity space and scale space. It helps the recovered stereo depth generate more details and avoid error matching from noisy appearance caused by sunlight or shadows etc..

We provide a CSPN perspective for spatial pyramid pooling (SPP), and propose a more effective SPP module, which further boosts performance for both depth completion and stereo.

The structure of this paper is organized as follows. We provide related work in Sec. 2. Sec. 3.1 elaborates the design, theoretical background of CSPN, and its variants including 3DCSPN and its relation with SPP. In Sec. 3.3.1 and Sec. 3.4, we present the details about how we adapt CSPN to depth completion and stereo matching correspondingly. Finally, we evaluate the results of our algorithms on all the tasks quantitatively and qualitatively in Sec. 4.

Related Work

Depth estimation has been a center problem for computer vision and robotics for a long time. Due to space limitation, here we summarize those works in several most relevant aspects.

Single view depth estimation via CNN and CRF. We first review single view depth estimation since methods in this area motivated our design. Deep neural networks (DCN) provide strong feature representation for various vision tasks. Depending on DCN, numerous algorithms are developed, including supervised methods ,, semi-supervised methods or unsupervised methods . Others tried to improve the estimated details further by appending a conditional random field (CRF) and multi-task correlation with joint training . However, the affinity for measuring the coherence of neighboring pixels is commonly manual-designed based on color similarity or intervening contour with RBF kernel . In our case, CSPN learn data dependent affinity that has demonstrated to be more robust in practice.

Depth Enhancement. Traditionally, depth output can be also efficiently enhanced with explicitly designed affinity through image filtering , or data-driven ones through total variation (TV) and learning to diffuse by incorporating more priors into diffusion partial differential equations (PDEs). However, due to the lack of an effective learning strategy, they are limited for large-scale complex visual enhancement.

Recently, deep learning based enhancement yields impressive results on super resolution of both images and depths . The network takes low-resolution inputs and output the high-resolution results, and is trained end-to-end where the mapping between input and output is implicitly learned. However, these methods are trained and experimented only with perfect correspondent ground-truth low-resolution and high-resolution depth maps and often a black-box model. In our scenario, neither the input nor ground truth depth are non-perfect, e.g., depths from a low cost LiDAR or a network, thus an explicit diffusion process to guide the enhancement such as SPN is necessary.

Learning affinity for spatial diffusion. Learning affinity matrix with deep CNN for diffusion or spatial propagation receives high interests in recent years due to its theoretical supports and guarantees . Maire et al. trained a deep CNN to directly predict the entities of an affinity matrix, which demonstrated good performance on image segmentation. However, the affinity is followed by an independent non-differentiable solver of spectral embedding, it can not be supervised end-to-end for the prediction task. Bertasius et al. introduced a random walk network that optimizes the objectives of pixel-wise affinity for semantic segmentation. Nevertheless, their affinity matrix needs additional supervision from ground-truth sparse pixel pairs, which limits the potential connections between pixels. Chen et al. try to explicit model an edge map for domain transform to improve the performance.

The most related work is SPN , where the learning of a large affinity matrix for diffusion is converted to learning a local linear spatial propagation, yielding a simple yet effective approach for output enhancement. However, as mentioned in Sec. 1, depth enhancement commonly needs local context, especially when sparse depth samples are available for depth completion. It might not be necessary to update a pixel by scanning the whole image. In our experiments, the proposed CSPN is more efficient and provides much better results.

Depth completion with sparse samples. Different from depth enhancement, the provided depths are usually from low-cost LiDAR, yielding a map with valid depth in only few hundreds of pixels, as illustrated in Fig. 1(a). Most recently, Ma et al. propose to treat the sparse depth map as an additional input to a ResNet based depth predictor, producing superior results than the depth output from CNN with sole image input. However, the output results are still blurry, and do not satisfy our requirements of depth as discussed in Sec. 1. In CSPN for depth completion, we embed the sampled depth samples in the diffusion process, where all the requirements we proposed are held properly.

Some other works directly convert 3D points from LiDAR to dense ones without image input , In these methods, the density of valid depth must be high enough to reveal the scene structure. Other works complete depth with an RGB image in a manner of in-painting, where dense depth values are partially available. These approaches are dealing with a different setup from ours.

Stereo depth estimation. Stereo depth estimation has long been a central problem in computer vision. Traditionally, Scharstein and Szeliski provide a taxonomy of stereo algorithms including: matching cost calculations, matching cost aggregation, disparity calculation and disparity refinement.

CNNs were first introduced to stereo matching by Zbontar and LeCun to replace the computation of the matching cost. Their method showed that by using CNNs, the matching could be more robust, and achieved SOTA results over KITTI Stereo benchmarks. However, the networks are still shallow, and it needs post-processing for refinement. Following , several methods were proposed to increase computational efficiency , or matching cost accuracy with stronger network and confidence predictions. Later, some works focused on post-process by incorporating high-level knowledge from objects such as Displets .

This inspires the study of stereo matching networks to develop a fully learnable architecture without manually designed processing. FlowNet are designed to find 2D optical flow by inserting two corresponding frames, which can be easily extended to stereo matching by limiting the searching within the epipolar line. PWCNet follows a similar idea while having cost volumes calculated using a pyramid warping strategy within a local region with size of $d\times d$. However, for stereo estimation, one may only consider a limited range for disparity matching based on the epipolar constraint. Therefore, to better model per-pixel disparity matching, GCNet proposes to generate a 3D cost volume of size $d\times h\times w\times c$ by densely comparing the feature at pixel (i, j) from the reference image to all possible matching pixels within the epipolar line at the target image. The network can estimate the best matching disparity through a $soft-argmin$ operation. PSMNet , embracing the experience of semantic segmentation studies, which additionally exploits scale space through pyramid spatial pooling and hourglass networks to capture global image context, yielding better results than GCNet. As can be seen, both GCNet and PSMNet are benefited from exploring a new dimension, i.e., disparity value space and scale space respectively, which motivates us to extend CSPN to 3D. Built upon PSMNet, 3D CSPN considers modeling the relationship with diffusion along their proposed new dimension, and produces more robust results.

Spatial pyramid for hierarchical context. As we also explore scale space for a dense prediction model, we would like to review spatial pyramid pooling (SPP) to provide more insight for our proposed model. Liu et al. first propose SPP to increase the empirical receptive field of a fully convolutional network. Such an idea is demonstrated to be very effective in both semantic segmentation , and depth estimation . Here, in our perspective, the parameters for SPP form a scale space that is manually set and experimentally determined based on certain dataset by previous works . However, our CSPN with 3D convolution can learn the affinity for fusing the proposed scale space, which softly discovers the proper scale of context for the network. Most recently, Xu et al. also propose to learn attention guided multi-scale CRF for depth estimation, which shares similar spirits with ours. We show in our experiments, such a strategy effectively improves the depth estimation results over PSMNet. In the near future, we will extend this idea to semantic segmentation to validate its generalization capability.

Our Approach

In this section, we first introduce the CSPN module we proposed, which is an anisotropic diffusion process and the diffusion tensor is learned through a deep CNN directly from the given image. Then we describe 3DCSPN, and how to use it together with SPP. Finally, we elaborate on how we applied these modules to the depth completion and stereo depth estimation.

Given one depth map $D_{o}\in\textbf{R}^{m\times n}$, and one image ${\mathbf{X}}\in\textbf{R}^{m\times n}$, our task is to update the depth map to a new depth map $D_{n}$ within $N$ iteration steps, which not only reveals more structure details, but also improves the per-pixel depth estimation results.

Fig. 2(b) illustrates our updating operation in 2D. Formally, without loss of generality, we can embed the depth map $D_{o}\in\textbf{R}^{m\times n}$ to some hidden space ${\mathbf{H}}\in\textbf{R}^{m\times n\times c}$, where $c$ is the number of feature channels. The convolutional transformation functional with a kernel size of $k$ for each time step $t$ could be written as,

where the transformation kernel $\hat{\kappa}_{i,j}\in\textbf{R}^{k\times k\times c}$ is the output from an affinity network, which is spatially dependent on the input image. The kernel size $k$ is usually set as an odd number so that the computational context surrounding pixel $(i,j)$ is symmetric. $\odot$ means element-wise product here. Here, following SPN , we normalize kernel weights to the range of $(-1,1)$ so that the model can be stabilized when the condition $\sum_{a,b,a,b\neq 0}|\kappa_{i,j}(a,b)|\leq 1$ is satisfied. Finally, we perform $N$ iterations to reach a stable status.

Correspondence to diffusion process with a partial differential equation (PDE). Similar to , here we show CSPN holds the desired property such as model stability of SPN. Formally, we can rewrite the propagation in Eq. (1) as a process of diffusion evolution by first doing column-first vectorization of feature map ${\mathbf{H}}$ to ${\mathbf{H}}_{v}\in\textbf{R}^{mn\!\times\!c}$.

where $\lambda_{i,j}=\sum_{a,b}\kappa_{i,j}(a,b)$, and ${\mathbf{D}}$ is the degree matrix containing all the $\lambda_{i,j}$, and ${\mathbf{A}}$ is the affinity matrix. The diffusion process expressed as a partial differential equation (PDE) is derived as:

Therefore, same as SPN , the stability of CSPN can be guaranteed if the norm of the temporal Jacobian is equal to or less than one. In our case, we follow their proof using the Gershgorin’s Theorem . Formally, taking Eq. (3), for any $t$ greater than 1, we have,

where $\lambda_{max}$ is the maximum Eigenvalue of ${\mathbf{A}}$. This satisfies the model stability condition.

In our formulation, different from which scans the whole image in four directions sequentially (Fig. 2(a)), CSPN propagates a local area towards all directions simultaneously at each step (Fig. 2(b)), i.e., with $k\!\times\!k$ local context. Larger context can be observed when recurrent processing is performed, and the context acquiring rate is in the order of $O(kN)$. In practice, we choose to use convolutional operation due to that it can be implemented through image vectorization, yielding efficient depth refinement.

In principal, CSPN could also be derived from mean-field approximation for solving a special type of locally connected conditional random field (CRF) model , where we rely the long range dependency on the propagation and the deep neural network. However, since our approach adopts more efficient linear propagation and can be regarded as a special case of pairwise potential in graphical models, to be more accurate description, we call our strategy convolutional spatial propagation in the field of diffusion process.

Complexity analysis. As formulated in Eq. (1), our CSPN is highly parallelizable and GPU-friendly. Under the assumption of the unlimited number of computing cores for convolution, the theoretical complexity of using CUDA with GPU for one step CSPN is $O(\log_{2}(k^{2}))$, where $k$ is the kernel size. In theory, convolution operation can be performed in parallel for all pixels and channels, which has a constant complexity of $O(1)$ . Therefore, performing $N$-step propagation, the theoretical overall complexity for CSPN is $O(\log_{2}(k^{2})N)$, which is independent of image size $(m,n)$.

SPN adopts scanning row/column-wise propagation in four directions. Using $k$-way connection and running in parallel, the complexity for one step is $O(\log_{2}(k))$. The propagation needs to scan the full image from one side to another, thus the complexity for SPN is $O(\log_{2}(k)(m+n))$. Though this is already more efficient than the densely connected CRF , whose implementation complexity with permutohedral lattice is $O(mnN)$, ours $O(\log_{2}(k^{2})N)$ is more efficient since the number of iterations $N$ is always much smaller than the size of image $m,n$.

We elaborate on the practical time cost in our experiments (Sec. 4) with a Titan X GPU. For example, with $k=3$ and $N=12$, CSPN runs 6x faster than SPN, and is $30\%$ more accurate, demonstrating both efficiency and effectiveness of the proposed approach.

2 CSPN variants for performance boosting

In this section, we present the two extensions of CSPN, i.e., 3DCSPN and its conjunction with SPP.

3DCSPN. As introduced in Sec. 1, we extend CSPN to 3D for processing the 3D cost volume that are commonly used for stereo estimation . We illustrate 3DCSPN in Fig. 2(c). Similar to Eq. (1), given a 3D feature volume ${\mathbf{H}}\in\textbf{R}^{d\times m\times n\times c}$, where d is an additional feature dimension, the formation for 3D CSPN could be written as,

which simply adds a new dimension for propagation comparing to Eq. (1), and we can see the original theoretical properties are all well maintained.

Convolutional spatial pyramid fusion (CSPF). The second module in the architecture we enhanced is the spatial pyramid pooling (SPP) as illustrated in Fig. 3(a). Here, we can see that each branch of SPP can be treated as a special case of one-step CSPN given proper kernel size and convolution stride. Formally, given a feature map with the size of $h\times w\times c$, and target pooled feature map with spatial size of $p\times q$, spatial pooling computes the mean value within each parted grid with size of $h/p\times w/q$. This is equivalent to one step CSPN (Eq. (1)) by setting both convolution kernel size and stride to be $h/p\times w/q$, and all the values in $\kappa(a,b)$ to be uniform. However, we know that features can be very different at impacting the final performance as shown in many attention models . Therefore, we propose to learn such a pooling/transformation kernel $\kappa(a,b)$ using CSPN for this SPP module. As shown in Fig. 3(b), in our case, we output an affinity matrix from the same feature block for spatial pooling, based on which we do one step 2D CSPN, yielding the required pooled feature map with size of $p\times q$. Specifically, feature maps with target sizes of 64 x 64, 32 x 32, 16 x16 and 8 x 8 are adopted (Fig. 3(a)), and all the feature maps share the same network output for computing the pooling kernels. In practice, our network outputs a one channel weight map with size of $h\times w\times 1$, and for each target size of pooled feature, we first partitioning the weight map to pooling regions, and compute the pooling/transformation kernel $\kappa()$ within each region. However, rather than using the normalization in Eq. (1), we enforce the kernel value to be positive, and normalize the them over all output values without dropping the kernel value of center pixel, which is formally written as, $\kappa_{i,j}(a,b)=\frac{|\hat{\kappa}_{i,j}(a,b)|}{\sum_{a,b}|\hat{\kappa}_{i,j}(a,b)|}$. Here, the output weight map is repeated $c$ times to match the feature dimension in CSPN. We call our strategy of multi-scale feature computation as convolutional spatial pyramid pooling (CSPP) to simplify our description later.

Last, we need to fuse the $l$ feature maps from all the layers of the spatial pyramid. Rather than directly concatenating all the pooled features into a feature map with size $h\times w\times lc$ as usual , we adopt the strategy of weighted averaging, which is illustrated in Fig. 3(c). Specifically, we concatenate the output spatial pyramid features into a 4D feature volume with size $l\times h\times w\times c$, and learn a transformation kernel with size of $l\times 3\times 3$. After doing one step 3D CSPN without padding in the layer dimension, i.e., use padding size of $$, we obtain a fused feature map with size$h\times w\times c$. Here, we also adopt the same kernel normalization as that in CSPP. Here, we use one independent branch for computing the transformation kernel for each layer of spatial pyramid, and concatenate them to a 3D CSPN kernel. We call this strategy as convolutional feature fusion (CFF) to simplify our description. As shown in Fig. 3(d), our final spatial pooling strategy is a combination of CSPP and CFF, which we call convolutional spatial pyramid fusion (CSPF). CSPF produces significant performance boost over the original SPP module in our tasks, especially for stereo depth estimation as demonstrated in Sec. 4.2.

Finally, we also adopt Atrous SPP (ASPP) to replace SPP for multi-scale feature pooling without feature size reduction. Specifically, ASPP use dilated convolution to obtain features within various context. Similarly, our CSPN can be performed in the same manner of dilated convolution by learning a spatial dependent transformation kernel for each layer of the pyramid. Therefore, we also extend ASPP to atrous CSPP (ACSPP) for computing spatial pyramid features. In our experiments, we use the set of dilation rates proposed in ASPP including 6 x 6, 12 x 12, 18 x18 and 24 x 24, and found ACSPP achieves better performance than that from CSPP. Combining it with CFF, called ACSPF, yields our best setting for competing over the benchmarks.

3 Learning depth completion with CSPN

In depth completion, we have a sparse depth map $D_{s}$ (Fig. 4) joint with an RGB image as inputs to our model. Specifically, a sparse set of pixels already have known depth values (obtained by other sensors). These pixels are used to guide the depth estimation for the remaining pixels. As shown in Fig. 4, our architecture is similar to that from Ma et al. , but with three major modifications: a ACSPF module (red block), a CSPN layer with sparse depths (orange block) and mirror connections. Later, we will elaborate the motivation and design of the second and third modifications.

As stated in Sec. 3.1, CSPN help to recover better details. However, for depth completion, we need to preserve the depth values at those valid pixels with known depth. To achieve this goal, we modify CSPN to include the sparse depth map $D_{s}$ in the diffusion process. Specifically, we embed $D_{s}$ to a hidden representation ${\mathbf{H}}^{s}$, and re-write the update equation of ${\mathbf{H}}$ by adding a replacement step after performing Eq. (1),

where $m_{i,j}=\textbf{I}(d_{i,j}^{s}>0)$ is an indicator for the validity of sparse depth at $(i,j)$.

From Eq. (6), the updating still follows the diffusion process with PDE, where the affinity matrix can be built by simply replacing the rows satisfying $m_{i,j}=1$ in ${\mathbf{G}}$ (Eq. (2)) with ${\mathbf{e}}_{i+j*m}^{T}$. Here ${\mathbf{e}}_{i+j*m}$ is an unit vector with the value at $i+j*m$ as 1. Therefore, the summation of each row is still $1$, and the stabilization still holds in this case.

With CSPN, we propagate the information from those sparse depth to its surrounding pixels such that the smoothness between the sparse depths and their neighbors are maintained, and thanks to the learned image dependent affinity, the final depth map is well aligned with image structures.

Our strategy has several advantages over the previous state-of-the-art sparse-to-dense methods . In Fig. 5(a), we plot a histogram of depth displacement from ground truth at given sparse depth pixels from the output of Ma et al. . It shows that the original values of sparse depth points cannot be preserved, and some pixels could have very large displacement (0.2m), indicating that directly training a CNN for depth prediction does not preserve the value of real sparse depths provided. To ensure such a property, one may simply replace the depths from the outputs withthe original values at these pixels with known depth, however, this yields non-smooth depth gradient w.r.t. surrounding pixels. In Fig. 6(c), we plot such an example, at right of the figure, we compute Sobel gradient of the depth map along x direction, where we can clearly see that the gradients surrounding pixels with replaced depth values are non-smooth. We statistically verify this in Fig. 5(b) using 500 sparse samples, the blue bars are the histogram of gradient error at sparse pixels by comparing the gradient of the depth map with sparse depth replacement and of ground truth depth map. We can see the difference is significant, 2/3 of the sparse pixels has large gradient error. Our method, on the other hand, as shown with the green bars in Fig. 5(b), the average gradient error is much smaller, and most pixels have no error at all. In Fig. 6(d), we show the depth gradients surrounding sparse pixels are smooth and close to ground truth, demonstrating the effectiveness of our propagation scheme.

3.2 Mirror connections for better details

In CSPN, in order to recover spatial details in the input image, the network for learning affinity matrix should contain less spatial down-sampling. Therefore, Liu et al. build a separate deep network for affinity learning. In our case, we wish to share the feature extractors for affinity and depth estimation, which saves us both memory and time cost for learning and inference. Therefore, as shown in Fig. 4, we fork an additional output from the given depth network to predict the affinity matrix.

However, spatial information is weakened or even lost with the down sampling operation during the forward process of the ResNet in . Thus, we add mirror connections similar with the U-shape network by directly concatenating the feature from encoder to up-projection layers as illustrated by “UpProj$\_$Cat” layer in Fig. 4. Notice that it is important to carefully select the end-point of mirror connections. Through experimenting three possible positions to append the connection, i.e., after conv, after bn and after relu as shown by the “UpProj” layer in Fig. 4 , we found the last position provides the best results in the NYU v2 dataset (Sec. 4.1.2). In this configuration, we found not only the depth output from the network is better recovered, but also the results after CSPN is further refined. Finally we adopt the same training loss as , yielding an end-to-end learning system.

4 Learning stereo matching with CSPN

In this application, we adopt a network architecture similar to PSMNet as shown in Fig. 7. The left and right images are fed to two weight-sharing CNN, yielding corresponding feature maps, a spatial pooling module for feature harvesting by concatenating representations from sub-regions with different sizes. The two produced feature maps are then used to form a 4D cost volume, which is fed into a 3D CNN for disparity regression. We refer readers to the original paper for more details due to space limitation. Here, we made two changes by replacing CSPF with their spatial pooling module and appending 3DCSPN after multi-scale outputs. Here, we update the spatial pooling and 3D module with our proposed CSPF and 3DCSPN (Sec. 3.2), it is straight-forward to use CSPF and we describe the details of applying 3DCSPN in the following.

In Fig. 8, we zoom into the 3D module of Fig. 7 to clarify the 3D CSPN we applied for disparity regression. In PSMNet, three predicted disparity volumes with size of $d/4\times h/4\times w/4\times 1$ are output at different stages from a stacked hourglass network. Here $d,h,w$ is the maximum disparity, height and width of the input image correspondingly. Similar to the appending strategy of 2D CSPN for single image depth prediction in Sec. 3.3.2, after the disparity volume at each stage, we append a 3D CSPN with kernel size $k\times k\times k$ to combine the contexts from neighbor pixels, where the affinity matrix is learned from the same feature block as the outputs. Then, trilinear upsampling is applied to upsample a disparity volume to $d\times h\times w\times 1$ for disparity map regression, yielding an output with shape of $h\times w\times 1$. Here, 3DCSPN finish processing over the disparity space (ds).

To fuse the multiple disparity maps output from different stages, PSMNet manually sets the weight to average the outputs. In our case, we concatenate them into a 4D volume with size $s\times h\times w\times 1$, where $s=3$ is the number of disparity maps. Similarly, we can perform a 3D CSPN with kernel size as $s\times k\times k$ to connect the multi-stage predictions, which is conceptually similar as attention models for multi-scale feature fusion . Last, we use feature padding with size of $$, so that the first dimension is reduced to$1$with one iteration, and we obtain a single regressed disparity map with shape$h\times w\times 1$ for final depth estimation. Here, 3DCSPN finish processing over the scale space (ss).

Finally, for training the full network, We use the same $soft$-$argmin$ disparity regression method proposed by GCNet to convert final discretized disparity values to continuous ones.

Then, the continuous disparity value is compared against the ground truth disparity value using the $L_{1}$ loss. Formally, the loss function is defined as:

where $d^{*}$ is a ground truth disparity, and $\widehat{d}$ is the predicted disparity from Eq. (7).

Experiments

In this section, we first describe our implementation details, the datasets and evaluation metrics used in our experiments, then present a comprehensive evaluation of CSPN and its variants, i.e., CSPF and 3D CSPN, on the two tasks of depth completion and stereo matching.

Implementation details. Following the network proposed in , the weights of ResNet in the encoding layers for depth estimation (Sec. 3.3.2) are initialized with models pretrained on the ImageNet dataset . Our models are trained with SGD optimizer, and we use a small batch size of 8 and train for 40 epochs for all the experiments, and the model performed best on the validation set is used for testing. The learning rate starts at 0.01 ,and will reduced to 20$\%$ when there is not any improvements in three consecutive epochs. A small weight decay of $10^{-4}$ is applied for regularization.

For depth, we found out that propagation with hidden representation ${\mathbf{H}}$ achieved marginal improvement over doing propagation within the domain of depth $D$. Therefore, we perform all our experiments directly with $D$ rather than learning an additional embedding layer. For sparse depth samples, we adopt 500 sparse samples as that is used in .

All our experiments are evaluated on two datasets: NYU v2 and KITTI Odometry , using commonly used metrics.

NYU v2. The NYU-Depth-v2 dataset consists of RGB and depth images collected from 464 different indoor scenes. We use the official split of data, where 249 scenes are used for training and we sample 50K images out of the training set with the same manner as . For testing, following the standard setting , the small labeled test set with 654 images is used the final performance. The original image of size $640\!\times\!480$ are first downsampled to half and then center-cropped, producing a network input size of $304\!\times\!228$.

KITTI odometry dataset. It includes both camera and LiDAR measurements, and consists of 22 sequences. Half of the sequence is used for training while the other half is for evaluation. Following , we use all 46k images from the training sequences for training, and a random subset of 3200 images from the test sequences for evaluation. Specifically, we take the bottom part $912\!\times\!228$ due to the lack of depth at the top area, and only evaluate the pixels with ground truth.

Metrics. We adopt the same metrics and use their implementation in . Given ground truth depth $D^{*}=\{d^{*}\}$ and predicted depth $D=\{d\}$, the metrics include: (1) RMSE: $\sqrt{\frac{1}{|D|}\sum_{d\in D}||d^{*}-d||^{2}}$. (2) Abs Rel: $\frac{1}{|D|}\sum_{d\in D}|d^{*}-d|/d^{*}$. (3) $\delta_{t}$: $\%$ of $d\in D$, s.t. $max(\frac{d^{*}}{d},\frac{d}{d^{*}})<t$, where $t\in\{1.25,1.25^{2},1.25^{3}\}$. Nevertheless, for the third metric, we found that the depth accuracy is very high when sparse depth is provided, $t=1.25$ is already a very loose criteria where almost $100\%$ of pixels are judged as correct, which can hardly distinguish different methods as shown in (Tab. I). Thus we adopt more strict criteria for correctness by choosing $t\in\{1.02,1.05,1.10\}$.

1.2 Ablation study for CSPN Module

Here, we evaluate various hyper-parameters including kernel size $k$, number of iterations $N$ in Eq. (1) using the NYU v2 dataset for single image depth estimation. Then we provide an empirical evaluation of the running speed with a Titan X GPU on a computer with 16 GB memory.

Number of iterations. We adopt a kernel size of $3$ to validate the effect of iteration number $N$ in CSPN. As shown in Fig. 9(a), our CSPN has outperformed SPN (horizontal line) when iterated only four times. Also, we can get even better performance when more iterations are applied in the model during training. From our experiments, the accuracy is saturated when the number of iterations is increased to $24$, which we adopt for final evaluation.

Size of convolutional kernel. As shown in Fig. 9(b), larger convolutional kernel has similar effect with more iterations, due to the larger context being considered for propagation at each time step. Here, we hold the iteration number to $N=12$, and we can see the performance is better when $k$ is larger while saturated at size of $7$. We notice that the performance drops slightly when the kernel size is set to $9$. This is because that we use a fixed number of epoch, i.e., 40, for all the experiments, while larger kernel size induces more affinity to learn in propagation, which needs more epoch of data to converge. Later, when we train with more epochs, the model reaches similar performance with kernel size of $7$. Thus, we can see using kernel size of $7$ with $12$ iterations reaches similar performance of using kernel size of $3$ with $20$ iterations, which shows CSPN has the trade-off between kernel size and iterations. In practice, the two settings run with similar speed, while the latter costs much less memory. Therefore, we adopt kernel size as $3$ and number of iterations as $24$ in our comparisons.

Concatenation end-point for mirror connection. As discussed in Sec. 3.3.2, based on the given metrics, we experimented three concatenation places, i.e., after conv, after bn and after relu by fine-tuning with weights initialized from encoder network trained without mirror-connections. The corresponding RMSE are $0.531$, $0.158$ and $0.137$ correspondingly. Therefore, we adopt the proposed concatenation end-point.

Running speed For testing, in Fig. 9(c), we show the running time comparison between the SPN and CSPN with kernel size as $3$. We use the author’s PyTorch implementation online. As can be seen, we can get better performance within much less time for large images, while is comparable when image size is small $320\times 240$. For example, four iterations of CSPN on one $1024\times 768$ image only takes 3.689 $ms$, while SPN takes 127.902 $ms$. In addition, the time cost of SPN is linearly growing w.r.t. image size, while the time cost of CSPN is independent to image size and much faster as analyzed in Sec. 3. In practice, however, when the number of iterations is large, e.g., “CSPN Iter 20”, we found the practical time cost of CSPN also grows w.r.t. image size. In principle, we can eliminate such a memory bottleneck by customizing a new operation, which will be our future work. Nevertheless, without coding optimation, even at high iterations with large images, CSPN’s speed is still twice as fast as SPN.

Tab. II shows training and inference time in NYU v2. Our model memory cost for the single batch of NYU v2 is about 12GB, which is larger than Ma et al. and comparable to SPN . The learning/inference time for each epoch with 50K images is 0.9 hours, which is comparable to SPN .

1.3 Comparisons

Baseline algorithms. We compare our method against one of the SOTA algorithms for depth completion, i.e., Sparse-to-Dense , and against the most related SOTA dense prediction refinement strategies, i.e., SPN , to validate our algorithm.

Specifically, we adopt exactly the same backbone as , and modify only the architecture with our mirror connection (MC) (Sec. 3.3.2), and add additional module of 2D CSPN as illustrated in Fig. 4. To compare against SPN , we use the same feature map for affinity prediction, while only switch the linear propagation strategy. For reproducing the baselines, we use released code online from the authors.

In the following, as shown in Tab. I $\&$ Tab. III, we compare various methods in two different settings. (1) Direct refine the depth map after the network output without including sparse depth samples in propagation, i.e., not preserve sparse samples (SD) in final outputs. (2) Refine the depth using both the network output and sparse depth samples, i.e., preserve SD. In both cases, we show our proposed approaches largely outperform the baselines, yielding SOTA results for depth completion with very sparse depth samples.

NYU v2. Tab. I shows the comparison results. Our baseline results are the depth output from the network of Ma et al. , which takes as input a RGB image and a sparse depth map. At the upper part of Tab. I we show that the results for depth refinement using network outputs only. At row “Bilateral”, we refine the network output from using bilateral filtering as a post-processing module with their spatial-color affinity kernel tuned on our validation set. Although the output depths snap to image edges, the absolute depth accuracy is dropped since the filtering over-smoothed original depths. At row “DenseCRF“, we show the results filtered with DenseCRF . Specifially, we first evenly discretized the depth value to $256$ labels, and then did post-processing with both spatial and color RBF kernels with tuned parameters. The estimated depths are significantly improved thanks to the designed affinity. At row “SPN”, we show the results filtered with SPN , using the author provided affinity network. Due to joint training, the depth is improved with the learned affinity, yielding both better depth details and absolute accuracy. Switching SPN to CSPN (row “CSPN”) yields significantly better results, e.g., accuracy of $\delta\leq$ 1.02 increases from 67.5$\%$ to 76.2$\%$, demonstrating its effectiveness. Finally, at the row “Mirror connection (MC)”, we show the results of just modifying the network with mirror connections as stated in Sec. 3.3.2. The results turn out to be even better than that from SPN and CSPN, demonstrating that by simply adding feature from the initial layers, depth can be better learned.

At the lower part of Tab. I, we show the results using both network outputs and sparse depth samples at later stage, so that all the results preserves the sparse depth value provided. We randomly select 500 depth samples per image from the ground truth depth map. At row “Replacement”, as illustrated in Fig. 6(c), we first directly replace the depth values at sparse depth sample locations, yielding reasonable improvements in depth accuracy. However, the visual results are not smooth, which is not satisfactory. In addition, we consider a baseline method using as-rigid-as-possible (ARAP) warping. Basically the input depth map is warped with the sparse depth samples as control points. At row “ARAP”, we show its results, which just marginally improves the estimation over the baseline network. For SPN, we also apply the similar replacement operation in Eq. (6) for propagation, and the results are shown at row “SPN”, which outperforms both the results form ARAP and SPN without propagation of SD due to joint training helps fix the error of warping. At row “MC + SPN”, we use our architecture with mirror connection for learning affinity with SPN, which outperforms “SPN”, while we did not see any improvements compared with that only using MC. Nevertheless, by replacing SPN with our CSPN, as shown in row “MC + CSPN”, the results can be further improved by a large margin and performs best in all cases. We think this is mostly because CSPN updates more efficiently than SPN during the training.

Finally, we validate the effect of our convolutional spatial pyramid fusion (CSPF) (Sec. 3.2). We first add ASPP module in middle part of the network (“MC+CSPN+ASPP”) to serve as the baseline, which slightly improves the results. Then we add convolutional feature fusion (CFF) (“MC+CSPN+ASPP+CFF”), and add repalce ASPP with ACSPP (“MC+CSPN+ACSPP”), showing both of the components proposed in CSPF are effective for improving the performance. Jointly using ACSPP and CFF (“MC+CSPN+ACSPF”) yields the best performance. This is majorly because larger context and better spatial details are obtained.

We provide more complete ablation validation of CSPF in Sec. 4.2 due to PSMNet also adopts SPP in their architecture. Some visualizations are shown in Fig. 10. We found the results from CSPN and CSPF do capture better structure from images (highlighted with dashed bounding boxes) than that from other state-of-the-art strategies.

KITTI. Tab. III shows the depth refinement with both color and sparse depth samples. Ours final model “MC + CSPN + ACSPF” largely outperforms other SOTA strategies, which shows the generalization of the proposed approach. For instance, with a very strict metric $\delta<1.02$, ours improves the baseline from $30\%$ to $72\%$. More importantly, CSPN is running very efficiently, thus can be applied to real applications. Some visualization results are shown at the bottom in Fig. 11. Compared to the network outputs from and SPN refinement, CSPN sees much more details and thin structures such as poles near the road (first image (f)), and trunk on the grass (second image (f)). For the third image, we highlight a car under shadow in the left, whose depth is difficult to learn. We can see SPN fails to refine such a case in (e) due to globally varying lighting variations, while CSPN learns local contrast and successfully recover the silhouette of the car. Finally, by adding “ACSPF” in the framework, the spatial reduction of last block in ResNet encoder is removed (as shown in Fig. 4). Therefore, the results are further improved, and better local details are recovered. Finally, we also submit our results to the KITTI depth completion challenge http://www.cvlibs.net/datasets/kitti/eval_depth.php?benchmark=depth_completion and show that our results is better than previous SOTA method at the time of submission.

2 Stereo depth estimation

Implementation details. The base network we adopted is from the PSMNet , and we follow the same training strategies. Specifically, for learning CSPN, we adopt Adam optimizer with $\beta_{1}$ = 0.9, $\beta_{2}$ = 0.999, and batch size is set to 16 for training on eight Nvidia P40 GPUs(each of 2). We performed color normalization on the entire dataset of Scene Flow for data preprocessing. During training, we crop the input image to $512\times 256$. We first train our network from scratch on Scene Flow dataset for 10 epochs, and the learning rate during this periods is set to $0.001$. When train on KITTI, we finetune the model obtained from Scene Flow for another 600 epochs. The learning rate starts from $0.001$ and decrease 10% each 200 epochs. Acquired by accumulating Velodyne HDL-64 Laser scanner, KITTI stereo ground truth is relatively sparse, and we only compute the loss where LiDAR ground truth is available.

we evaluate our method on following datasets: Scene Flow , KITTI Stereo 2012 , KITTI Stereo 2015 .

Scene Flow. A large scale dataset contains 35454 training and 4370 test stereo pairs in 960x540 pixel resolution, rendered from various synthetic sequences. Pixels besides our max disparity are excluded in loss function.

KITTI Stereo 2012. A real-world dataset with street views from a driving car, consists of 194 training and 195 test stereo pairs in 1240x376 resolution. Ground truth has been aquired by accumulating 3D point clouds from a 360 degree Velodyne HDL-64 Laserscanner. We divided the whole training data into 160 training and 34 validate stereo pairs, we adopted color image as network input in this work.

KITTI Stereo 2015. Compared to KITTI 2012, KITTI 2015 consists of 200 training and 200 test stereo pairs in 1240x376 resolution. Also, it comprises dynamic scenes for which the ground truth has been established in a semi-automatic process. We further divided the whole training data into 160 training and 40 validate stereo pairs.

Metrics. Since different datasets have various metrics for comparison, we list the corresponding evaluation metric as follows,

Scene Flow: the end-point error (EPE) is used. Formally, the difference could be written as $EPE(d^{*},\hat{d})=\|d^{*}-\hat{d}\|_{2}$.

KITTI 2012 and 2015: the percentages of erroneous pixels. Specifically, a pixel is considered to be an erroneous pixel when its disparity error is larger than $t$ pixels. Then, the percentages of erroneous pixels in non-occluded (Out-Noc) and all (Out-All) areas are calculated. Specifically, for benchmark 2012, $t\in\{2,3,4,5\}$. While for benchmark 2015, a pixel is considered to be wrong when the disparity error is larger than $3$ pixels or relatively $5\%$, whichever is less stringent. In addition, results on both left image (D1-All) and right image (D2-All) are evaluated. We refer the reader to their original page http://www.cvlibs.net/datasets/kitti/eval_stereo_flow.php?benchmark=stereo for more detailed information about other evaluated numbers. Here we only list the major metric to rank different algorithms.

2.2 Ablation Study

We do various ablation studies based on the Scene Flow dataset to validate each component of our networks as shown in Fig. 8 and Fig. 3. We first train a baseline results with the code provided online by the author of PSMNet https://github.com/JiaRenChang/PSMNet.

Study the 3D module. We first evaluate the components proposed in our 3D module (Fig. 8) in Tab. IV. In order to show that propagation in the new dimension benefits the results, we first adopt 2D CSPN for depth refinement as proposed for single image depth refinement over the three 2D disparity maps using the affinity predicted from the same feature, i.e., “CSPN”. As expected, it reduces the EPE error from 1.119 to 0.992. Then, we switch the 2D CSPN to 3D CSPN as proposed in Sec. 3.4, i.e., “3DCSP_ds”, the results are further improved to 0.971. Here, the footnote “ds” is short for disparity space, indicating the 3DCSPN is performed over the disparity outputs with shape $d\times h\times w\times 1$. “3DCSPN_ss” shows the results by using 3D CSPN over the space for multi-stage outputs fusion, which also helps the performance from our baseline. Jointly using the two 3D CSPNs, i.e., “3DCSPN_ds_ss”, yields the best result, outperforming our baseline method by a large margin. At last row, “3DCSPN_ds_ss + ACSPF” shows the results of combining our 3D module with our enhanced ASPP module together, which additionally reduce the error around $30\%$ w.r.t. to the baseline.

Study the CSPF module. Here, we evaluate different components for enhancing the SPP module that is also adopted in PSMNet, as shown in Fig. 3. For all the variations, we adopt “3DCSPN_ds” as our 3D module for ablation study. As introduced in Sec. 3.2, “CSPP” means we use 2D CSPN over the spatial pooling grid, which reduces the EPE error from 0.971 to 0.954. We then study another spatial pooling strategy with dilated convolution, i.e., “ASPP”, which produces similar performance as SPP. Surprisingly, as shown in row “ACSPP”, jointly using our 2D CSPN with ASPP produces error much smaller (0.902) than that with SPP (0.954). At row “CFF”, we use out proposed fusion strategy to combine the pooled features from the spatial pyramid, which also significantly improves over the SPP baseline, reducing EPE error from 0.954 to 0.905. Finally, combining ACSPP and CFF, i.e., “ACSPF”, yields the best performance, which is selected as our final SPP module. Fig. 12(c) shows a few examples of the output from the Scene Flow dataset, and we can see the predicted results are very close to ground truth, which are exceptionally good in handling detailed object structures.

2.3 Comparisons

Baseline algorithms. To validate the algorithm, in addition to comparing over the Scene Flow test set, we also submitted our results to KITTI 2012 and 2015 test evaluation server to compare against other SOTA methods proposed in recent years, including PSM-Net , iResNet-i2 , GC-Net , EdgeStereo , SGMNet , Displets v2 and MC-CNN . For KITTI benchmarks, we choose the same maximum disparity value used in PSM-Net , which is 192.

As summarized in Table VI, our method outperforms all others methods by a notable margin (above relatively $10\%$), and performs the best over all the major metrics both in KITTI 2012 http://www.cvlibs.net/datasets/kitti/eval_stereo_flow.php?benchmark=stereo and 2015 http://www.cvlibs.net/datasets/kitti/eval_scene_flow.php?benchmark=stereo&eval_gt=all&eval_area=all. By checking detailed numbers in KITTI 2015, we are better at improving the static background than the foreground, which is reasonable because background has much larger amount of training pixels for learning the propagation affinity. Fig. 12 (a) and (b) show several examples by comparing our algorithm to the baseline method PSMNet over KITTI 2012 and 2015 respectively, and we mark out the improved regions with dashed bounding boxes. As can be seen, CSPN not only better recovers the over all scene structure, but also superior in recovering detailed scene structures. More results are available in the KITTI leaderboard pages.

Conclusion

In this paper, we propose an effective module, namely convolutional spatial propagation network (CSPN), for two depth estimation tasks, i.e., depth completion and stereo depth estimation. CSPN can be jointly learned with any type of depth estimation neural networks, and could be regarded as a linear diffusion process with guarantee of model stability. Comparing with previous spatial propagation network , CSPN is more efficient (2-5$\times$ faster in practice especially when the image is large), and more accurate (over $30\%$ improvement) in terms of depth completion.

In addition, we extend CSPN to 3D, namely 3D CSPN, and combine it with spatial pyramid pooling (SPP), namely CSPF. These modules are shown to be effective in the two tasks we focused. Last, we further tune CSPN to better adapt each task. For depth completion, we embed sparse depth samples into its propagation process, and for stereo matching, we allow 3D CSPN to diffuse along the dimension of both the disparity spaces and the scale space. For both tasks, our designed approaches with CSPN provides superior improvement over other SOTA methods . Since many components of our framework are general, in the future, we plan to apply them to other tasks such as image segmentation and image enhancement.

Acknowledgments

References