Adaptive Context-Aware Multi-Modal Network for Depth Completion

Introduction

Depth information is crucial for 3D vision tasks, e.g., 6D object pose estimation , 3D object detection , and human pose estimation . To complete these tasks, various depth sensors such as LiDAR have been invented to acquire depth information. However, current depth sensors are not able to obtain dense maps for outdoor scenes, which are essential in various applications, especially autonomous driving. Therefore, depth completion from sparse depth mapsThe sparse depth map is generated by projecting the LiDAR data to the image plane, and the value in locations without depth information is . and RGB images has attracted intensive attention.

Depth completion is a challenging problem because the depth values obtained by sensors are highly sparse and irregularly spaced. For example, in the KITTI dataset , there are only $5.9\%$ pixels with depth information obtained by the Velodyne HDL-64e (64 layers) LiDAR in the whole image space, as shown in Figure 1. Traditional methods rely on handcrafted features and global constraints on the output depth values, which are inaccurate. Recent studies have demonstrated great advantages of deep Convolutional Neural Networks (CNNs) on depth completion. By extending the convolutional operation with sparsity-invariance or introducing more geometric information , these deep methods can achieve way better performance than traditional methods.

In spite of the encouraging progress, existing depth completion methods suffer from a significant issue, which limits the depth completion performance. Specifically, the conventional convolutional operation applies kernels with regular structure (e.g., $3\times 3$) at all locations, which ignores the fact that the observed depth values are irregularly distributed in a sparse depth map and associates limited observed contexts for the unobserved, as shown in Figure 2. Thus, CNN-based methods are not adaptive to the pattern of observed spatial contextual information in a sparse depth map, resulting in a sub-optimal prediction of depth in unobserved locations.

To address this issue and further boost depth completion accuracy, we propose an Adaptive Context-Aware Multi-Modal Network (ACMNet, shown in Figure 3). Firstly, inspired by recent works on point cloud analysis , we model the observed contextual information adaptively by applying attention based graph propagation within multiple graphs constructed from observed pixels. Based on the efficient graph propagation, the model can associate the spatial context with observed depth values and then enhance the features of the unobserved pixels. To illustrate this, we provide a simple example in Figure 2. Compared to the sole convolutional operations, the proposed graph propagation (followed by a convolution) can make the unobserved pixels capture more related observed contextual information.

Furthermore, since we have multi-modality data, we need to reconsider the novel graph propagation in a multi-modal setting. Firstly, to better learn the relationship between observed pixels (nodes), we use the co-attention mechanism to propagate the multi-modal information of observed pixels in a symmetric structure. This step is conducted in the encoder to extract multi-scale and multi-modal features. However, this mechanism does not consider the fusion of multi-modal contextual information. A simple way to fuse the multi-modal data is by applying the simple concatenation or element-wise summation operation on the extracted feature maps, which was used by most of the existing works, e.g., . However, this type of fusion strategy cannot fully explore the heterogeneity of the two modalities. To address the issue, we further present the symmetric gated fusion strategy to combine the depth and RGB information in the decoder. In specific, the presented fusion strategy consists of two branches. One branch focuses on fusing the RGB information as supplementary into the depth information through learning an adaptive gating function, and the other one does the opposite. Therefore, each branch can maintain its own information and benefit from supplementary information from the other. Benefiting from the adaptive co-attention guided graph propagation and symmetric gated multi-modal feature fusion, our ACMNet is able to generate high-quality dense depth maps. In summary, our main contributions are:

We introduce the co-attention guided graph propagation to our depth completion network, which is adaptive to the sparsity patterns of sparse depth input and thus enables the unobserved pixels to capture useful observed contextual information more effectively.

To fuse the multi-modal contextual information efficiently, we further present the symmetric gated fusion strategy, which can learn the heterogeneity of the two modalities adaptively.

We demonstrate the effectiveness of ACMNet on two benchmarks, i.e., KITTI Depth Completion Dataset and NYU-v2 Dataset .

Related Work

Depth Completion. Traditional approaches solve the depth completion problem by formulating the task as an energy function optimization problem . However, these works showed some limitations in performance due to the employment of hand-crafted features.

Currently, CNNs have been a dominant solution for depth completion , outperforming traditional methods by a wide margin. In specific, to learn representations of the irregular and sparse LiDAR data, Uhrig et al. proposed the sparsity-invariant convolutional operation. Following this work, some variants of the sparse convolution are introduced . In the case of additional RGB data, Jaritz et al. showed that the late fusion strategy outperformed the early fusion. Ma et al. utilized self-supervised learning on sparse LiDAR data coupled with the stereo image pair to mitigate the need for ground truth dense depth. Yang et al. exploited the Conditional Prior Network to learn a depth prior on synthetic images. Additionally, there are also a bunch of works exploring other cues. For example, Zhang et al. trained a network to predict local surface normals for indoor scene depth completion, and later an extension for outdoor scenes was introduced in their latest work . Similarly, Xu et al. also explored the surface normal information to improve the performance by introducing a diffusion module. Cheng et al. proposed to learn affinities between adjacent pixels for the spatial propagation of the depth information. Following the two works, a recent work improved the propagation strategy through concentrating on the non-local neighbors and introducing a learnable affinity normalization. Inspired by the guided image filtering, Tang et al. designed a guided convolution module, which generates dynamic spatially-variant kernels using the image features, to extract the depth image features. In comparison, a recent work proposed to dynamically learn the filter by applying the Graph Neural Network (GNN) on the graph constructed from the predicted dense depth map. In contrast to these approaches, which paid little attention to the modelling of the multi-modal contexts, our work mainly aims at making unobserved pixels capture more useful observed contextual information from the input multi-modal data. Additionally, it is worth pointing out that although the latest work also exploits the graph models, there are many differences between it and ours. For example, it aims to consider the neighborhood relationship of the points in the 3D space through constructing a 3D graph from the dense depth map, which is obtained using a deep model. To arrive at this, it applies the dynamic kernel, which is learned through using a typical GNN model on the constructed graph, on the dense features at $1/8$ of original scale. In comparison, in this paper we study the propagation of the contexts with observed depth values at multiple scales in a multi-modal setting to enhance the features of the unobserved pixels.

Monocular Depth Estimation. From approaches based on probabilistic graphical models (e.g., MRFs) with hand-crafted features to the deep learning-based , the improvement of performance for monocular depth estimation has been pushed forward. Eigen et al. were the first to develop deep models for depth estimation. Following their work, a lot of supervised approaches have been proposed. However, these methods rely on large quantities of ground truth depth data, which is hard to acquire. To address this issue, Garg et al. and Godard et al. proposed to predict depth maps from stereo pair images by exploring unsupervised cues, while some recent works tried to utilize synthetic data based on the domain adaptation technique .

Graph-based Models. Conventional deep learning modules, such as CNNs, do not perform well on graphs. To model the graph data efficiently, Graph Models have been applied on various computer vision tasks, such as action recognition , point cloud analysis , few-shot image classification , and person re-identification . Graph Models are able to learn the representation of each target node by propagating its neighborhood information in a data-driven way and thus associate the contextual information. In this work, we design an attention-based graph propagation module and then extend it to the co-attention guided graph propagation for multi-modal data, which is capable of learning an efficient multi-modal representation for the input data through encouraging the adaptive contextual interactions.

Multi-modal Information Fusion. Multi-modal information fusion has been studied in various computer vision tasks, such as visual question answering , video action recognition , 3D object detection , and many more. A simple approach to fuse the multi-modal data is applying concatenation or summation operation into the input data or extracted feature maps . However, for a specific task, different modalities often provide different information, and therefore, the naive fusion strategy might fail to combine them effectively. To address this issue, some works, e.g., , proposed to exploit the attention mechanism to improve the performance. As for depth completion, current works mainly employed the naive fusion strategy. In fact, both naive strategy and attention based approaches fuse the multi-modal features in a single way, which is not enough to extract complementary information and then limits the performance. In contrast, we present the symmetric gated fusion strategy consisting of two fusion paths, each of which only focuses on one modal and extracts useful information adaptively from the other.

Our Approach

2 Network Architecture

Our overall network architecture follows a two-stream encoder-decoder fashion as previously , but with the improvement by integrating the novel CGPM and SGFM. We show the whole framework in Figure 3, and briefly explain the encoder and the decoder right here.

Encoder. The encoder targets learning discriminative multi-scale features from both the sparse depth and the RGB image. While researchers reached a consensus that standard convolutional operations can perform well in the image data, how to extract rich information from observed spatial contexts is still an open problem due to the extreme sparsity . In this paper, we show that the proposed CGPM has the potential to capture the related contextual information from the observed pixels with various patterns in an adaptive manner through learning dynamic weights of the relationship between adjacent nodes in the constructed graph. Specifically, our encoder consists of two conventional convolutional layers followed by a stack of CGPMs. The encoded features at each scale $\{{\bf F}_{S}^{l}\}_{l=1}^{L}$ and $\{{\bf F}_{I}^{l}\}_{l=1}^{L}$ can be computed as:

where $l=1,2,...,L$, and $f_{e}^{l}$ denotes the CGPM at level $l$, and ${\bf F}_{S}^{0}$ and ${\bf F}_{I}^{0}$ are the outputs of the beginning convolutional layers.

Decoder. The decoder aims to predict depth values of unobserved pixels in ${\bf X}_{S}$ given multi-scale and multi-modal features generated by the encoder mentioned above. To this end, one of the commonly studied problems is how to take full advantage of multi-modal representations. A straightforward idea is to directly concatenate or sum features progressively at different scales . However, as analyzed before, these naive fusion strategies fail to model the complementary information between multiple modalities satisfyingly. To alleviate the issue, we propose an adaptive symmetric gated fusion strategy to fuse the multi-modal contextual representations in a parallel structure. In specific, we design two parallel branches in the decoder, i.e., the depth and image branches. The depth branch preserves discriminative information of the sparse depth modality and meanwhile adaptively captures comprehensive information from the image model through learning dynamic gating weights, and vise versa for the image branch. The overall decoder architecture is described as follows.

As shown in Figure 3, at the beginning of the decoder, we feed ${\bf F}_{S}^{L}$ coupled with ${\bf F}_{I}^{L}$ into the first SGFM to generate the fused feature ${\bf Q}_{SI}^{L}$ and ${\bf Q}_{SI}^{L,\uparrow}$, which is acquired by up-sampling ${\bf Q}_{SI}^{L}$ through one deconvolutional layer. At the following levels $l$ from $L-1$ to , ${\bf Q}_{SI}^{l+1,\uparrow}$, ${\bf F}_{S}^{l}$ and ${\bf F}_{I}^{l}$ are fed into the SGFM at level $l$ together. Similarly, we can obtain the intermediate features ${\bf Q}_{IS}^{l}$ in the image branch. The procedure can be expressed as:

where $l=L-1,L-2,...,0$, and $f_{d}^{l}$ represents the SGFM.

Finally, we present two methods, i.e., end-integration and feature-integration, to combine the two branches to obtain the final recovered dense depth map, which will be described in detail in Sec. 3.5.

3 Co-Attention Guided Graph Propagation (CGPM)

The proposed CGPM is composed of a residual connection and a co-attention guided graph propagation module. First, we introduce the basic graph propagation module, which is employed in CGPM. In specificIn the following part, we deprecate the scale indexes $l$ to simplify our presentation in some cases., given the spatial position set $P=\{p_{0},p_{1},...,p_{n-1}\}$ of $n$ pixels with observed depth values, we define a graph $G(V,E)$, where $V$ is the vertex (or node) set corresponding to $P$, and $E\subseteq|V|\times|V|$ is the edge set. For a vertex $i$, we connect it to the $k$ nearest neighbour $\mathcal{N}_{i}$ according to the spatial locations. Note that, we build an individual graph for the CGPM at each scale. Thus, to obtain a specific $P^{l}$ at level $l$, which is in lower resolution, we generate ${\bf X}_{S}^{l}$ by applying max-pooling based down-sampling operation on ${\bf X}_{S}^{l-1}$. The graph’s construction process can be found in Figure 4. In the following, we first introduce the basic attention guided graph propagation component at level $l$ by taking the image stream as an example, then present the full CGPM.

Given the graph $G$ and the input feature maps ${\bf F}_{I}^{l-1}$, we expect to learn discriminative ${\bf F}_{I}^{l}$ by both adaptively encoding the contextual information of scenes and exploiting guidance for unobserved pixels from observed pixels. Specifically, we exploit two efficient stages, i.e., adaptive feature propagation within observed pixels and feature enhancement of unobserved pixels.

where $\alpha^{i,j}$ is the computed attentional weight, and ${\bf W}^{i,j}$ is the adaptive weight between neighboured nodes $i$ and $j$. Here, inspired by the works on point cloud, we exploit the self-attention mechanism to learn ${\bf W}^{i,j}$ adaptively by modelling the relationship between the connected nodes. Mathematically, the mapping function $f_{w}$ between ${\bf F}_{Io}^{\prime}$ and ${\bf W}^{i,j}$ can be expressed as:

where $[\cdot||\cdot]$ represents the concatenation operation, $\Delta{p^{i,j}}=p^{j}-p^{i}$ and $\Delta{{\bf F}_{Io}^{{}^{\prime}i,j}}={\bf F}_{Io}^{{}^{\prime}j}-{\bf F}_{Io}^{{}^{\prime}i}$ denote the spatial and feature distances between node $i$ and $j$, respectively. The $f_{w}$ is implemented by a two-layer MLP, the first one followed by one LeakyReLU activation function . Note that, permutation variant operations like convolution are not allowed here due to the unordered input. After obtaining ${\bf F}_{Io}^{\prime\prime}$, the features of unobserved pixels are enhanced by a standard convolutional operation. In addition, a residual connection is also utilized to preserve early information. We can use the same algorithm to conduct propagation in the depth stream.

As shown in Figure 3, in the CGPM in our encoder, we learn the adaptive weights ${\bf W}_{S}$ and ${\bf W}_{I}$ by considering both information from the image stream and the sparse depth stream, inspired by the co-attention mechanism . Therefore, in each CGPM, Eq. 4 can be re-written as:

4 Symmetric Gated Fusion (SGFM)

For obtained features ${\bf F}_{S}$ and ${\bf F}_{I}$, we develop an effective fusing strategy to adaptively absorb complementary information from the multi-modal contextual representations. For example, depth features encode the scene geometry structure, e.g., the distance from the camera to partial spatial locations. It contributes to inferring the depth of unobserved locations directly. In addition, RGB features contain semantic information and provide prior appearance knowledge of unobserved pixels. Instead of concatenating or summing them together directly with or without attention mechanism, we exploit the proposed SGFM with a symmetric structure, as shown in Figure 3. More specifically, at the beginning of the decoder, we employ the convolutional operation followed by a Sigmoid function on ${\bf F}_{S}^{L}$ to generate the adaptive gating weight ${\bf G}_{S}^{L}$. By applying the adaptive attention mechanism, the network can absorb meaningful information from the RGB branch and filter out the unrelated. Then we we feed the initial fused feature $[{\bf F}_{S}^{L}||{\bf G}_{S}^{L}\ast{\bf F}_{I}^{L}]$ into the Residual Block (abbr. ResBlock) to obtain the final fused features ${\bf Q}_{SI}^{L}$, which is then fed into a deconvolutional layer to generate ${\bf Q}_{SI}^{L,\uparrow}$. Therefore, the depth features can be improved by the complementary information automatically. At the other levels, there is a slight difference in learning the adaptive weights. In specific, at level $l\in\{L-1,L-2,...,0\}$, we learn the gating weights ${\bf G}_{S}^{l}$ using ${\bf Q}_{SI}^{l+1,\uparrow}$, rather than ${\bf F}_{S}^{l}$. Moreover, we feed the concatenated feature $[{\bf Q}_{SI}^{l+1,\uparrow}||[{\bf F}_{S}^{l}||{\bf G}_{S}^{l}\ast{\bf F}_{I}^{l}]]$ into the ResBlock at $l\in\{L-1,L-2,...,1\}$ or one convolutional layer at $l=0$ to get the fused feature. Due to the symmetry of the structure, a similar procedure is employed in the image branch. To illustrate the difference between the proposed fusion strategy and the existing ones, e.g., direct fusion and direct attention fusion, we provide the visual and quantitative comparisons among them in Figure 5 and the ablation study, respectively.

5 Branch Integration

By applying the proposed symmetric gated fusion modules, we obtain two sets of features, one from the image branch and the other from the depth branch. Here, we consider two methods, i.e., end-integration and feature-integration, to integrate them together and then obtain the final prediction result.

where $*$ represents the element-wise multiplication.

5.2 Feature-integration

Apart from the integration in the end, we can also combine the features extracted by the two branches. In specific, as shown in Figure 6, we fuse the intermediate features ${\bf Q}_{SI}$ and ${\bf Q}_{IS}$ through several convolutional operations to obtain ${\bf Q}_{F}$ progressively, and lastly obtain the final prediction $\hat{\bf Y}$ by applying one convolutional operation on the final integrated features.

6 Loss Function

The network is mainly driven by a masked mean squared error (MSE) loss between the ground truth semi-dense depth map ${\bf Y}$ and the prediction $\hat{\bf Y}$, which is defined as:

where $N_{s}$ denotes the number of pixels in the whole image space, and $\nabla$ represents first derivative along spatial directions. Finally, the full objective is:

where $\gamma_{1}$ and $\gamma_{2}$ are the trade-off factors, and are set to $0.5$ and $0.01$ in our experiments, respectively.

Experiments

In this section, we first introduce the datasets used in our experiments, and the implementation details. Then we evaluate our method by making comparisons against state-of-the-art methods. Finally, we conduct several ablations to analyze our framework.

KITTI Depth Completion Benchmark . It is currently the main benchmark for depth completion. The dataset consists of over $90,000$ frames with the ground truth semi-dense depth map for training and validation, and $1,000$ frames without the ground-truth for test. We train depth completion models on the training set, and then evaluate the performance on the official selected validation and test sets. During training, we crop all training data (images and depth maps, $375\times 1242$) to the size of validation and test data, i.e., $352\times 1216$. For evaluation, we adopt the official error metrics: root mean squared error (RMSE in $mm$, main metric for ranking), mean absolute error (MAE in $mm$), root mean squared error of the inverse depth (iRMSE in $1/km$), and mean absolute error of the inverse depth (iMAE in $1/km$).

NYU-v2 . This dataset consists of RGB and depth images collected from $464$ different indoor scenes. According to the official data split strategy, $249$ scenes are used for training, and $654$ labeled images are selected for evaluating the final performance . In our experiments, we sample around $48k$ images with annotations from the training set for training. Adopting similar experimental setting as , we firstly down-sample all images to half and center-crop them to $304\times 228$, and then sample $500$ sparse LiDAR points from the provided dense depth map randomly as the sparse depth data. We exploit root mean square error (RMSE in $meter$), mean absolute relative error (REL in $meter$), and the percentage of relative errors inside a certain threshold ($\delta_{t}$, $t\in\{1.25,1.25^{2},1.25^{3}\}$) as evaluation metrics.

2 Implementation Details

Graph Construction. For KITTI dataset, we build the graphs at three scales with $10000$, $5000$, and $2500$ observed pixels randomly sampled from the downsampled sparse depth maps, respectively, and we calculate $6$ nearest neighbours for each node. For NYU-v2, we randomly sample $250$, $125$, and $60$ points, respectively. Note that, we can create the graphs using either the 3D coordinates (e.g., camera coordinates) or the 2D coordinates (e.g., pixel coordinates). Here, we use the 3D coordinates, and we will study the differences in ablation studies.

Architecture Details. At each level of the encoder, we employ two CGPMs, and in the decoder, two ResBlocks are utilized in the symmetric gated fusion module at each scale. The feature channels in the modules are set to $64$. Our final results are obtained using the feature-integration, and in this case, we use two convolutional layers, each with $64$ output channels at each scale.

Training Details. We implement our depth completion framework in PyTorch. In specific, we optimize our network with the momentum of $\beta_{1}=0.9$, $\beta_{2}=0.999$, and the initial learning rate of $\alpha=0.0005$ using the ADAM solver . The model is trained for around $40$ epochs with a batch size of 8, and the learning rate is delayed by $0.5$ every $10$ epochs during training.

3 Comparison against the State-of-the-art

KITTI Dataset. In Table 1, we report the number of parameters as well as the performance of our approach and previous peer-reviewed works on KITTI depth completion benchmark. Note that, some of the existing approaches employ additional data during training. For example, DeepLiDAR renders $50K$ training samples using an open urban driving simulator to train the surface normal prediction network, and Certainty utilizes a pre-trained sematic segmentation model on Cityscapes as network initialization, which can provide high-level semantic information for depth completion. In contrast to these approaches, we train our network from scratch without any additional data. Nevertheless, our approach obtains a convincible improvement over most of the previous methods. In comparison to the latest works, i.e., CSPN++ and NLSPN , our model achieves very close performance, but our model has fewer parameters. Specifically, the RMSE errors of NLSPN and CSPN++ are $3mm$ and $1mm$ less than ours, respectively, but the number of their parameters is around four times larger than ours.

Figure 7 shows some qualitative results of ACMNet and other four state-of-the-art methods . Benefiting from our proposed co-attention guided graph propagation and symmetric gated fusion strategy, which exploit observed pixels’ information and capture the heterogeneity of the two modalities efficiently, ACMNet is capable of yielding high-performing dense depth map, preserving more details over boundary regions (e.g., the 2nd and 3rd examples), and performing better on the tiny/thin objects (the 1st example).

NYU-v2 Dataset. As shown in Table 2, most of latest works have close performance on this dataset. Our method performs better than almost all of methods except NLSPN , but as stated above the number of our model’s parameters is far less than it.

4 Ablation Study

Here, we conduct comprehensive ablation studies on KITTI selected validation dataset to verify the effectiveness of our proposed components. In following experiments, we set the channels of intermediate layers in networks to $32$ to speed up model training. Unless otherwise specified, we exploit the end-integration in most cases.

The effectiveness of the graph propagation. We first demonstrate the effectiveness of the proposed co-attention guided graph propagation by comparing the performance in four cases, i.e., (1) Baseline: no propagation used in the encoder and direct fusion in the decoder; (2) +GP: graph propagation in the encoder and direct fusion in the decoder; (3) +SG: no propagation in the encoder and symmetric gated fusion in the decoder; (4) +GP+SG: our whole model with the end-integration. As shown in Table 3, +GP and +GP+SG outperform Baseline and +SG, respectively, which demonstrates that the proposed graph propagation module better captures the spatial contextual information from sparse LiDAR data.

Furthermore, we carry out four additional experiments to analyze in which stage, such as the encoder (i.e., +GP and +GP+SG), decoder (referred as +GP/D and +GP/D+SG), or whole network (referred as +GP/W and +GP/W+SG), the graph propagation module performs better. As shown in Table 3, the comparisons (+GP v.s. +GP/D, and +GP+SG v.s. +GP/D+SG) indicate that applying the propagation module in the feature extraction stage is more effective in modeling the contextual information. Additionally, we can also observe that compared to +GP (+GP+SG), +GP/W (+GP/W+SG) causes some performance drop. This might be because in the decoder the structure of the observed pixels is not well-preserved after several operations in the encoder.

The effectiveness of the symmetric gated fusion. To verify that the proposed symmetric gated fusion strategy performs better than direct fusion, e.g., concatenation with or without attention (referred as DAF and DF, respectively), we compare six models, i.e., DF (namely Baseline), DAF, SG, GP+DF (namely Baseline+GP in Table 3), GP+DAF, and GP+SG. As shown in Table 4, SG outperforms both DAF and DF, demonstrating that the proposed symmetric gated fusion strategy is capable of combining the multi-modal information more effectively. Moreover, the comparisons between GP+SG, GP+DAF, and GP+DF can further support this conclusion.

Analysis of graph construction. Here, we investigate the impacts of three factors involved in constructing graphs. Note that, we conduct the following experiments using our final model with the end-integration. We report the results in Table 5.

Firstly, since we aim at capturing more observed multi-modal information to enhance the features of unobserved pixels by finding their spatial neighbours, it is interesting to explore the selection of the coordinate system, i.e., pixel coordinate system or camera coordinate system. In specific, for a set of observed pixels, we can construct a graph according to their 2D coordinates $\{(u_{i},v_{i})\}_{i=0}^{n-1}$ directly or 3D coordinates $\{(x_{i},y_{i},z_{i})\}_{i=0}^{n-1}$, which are obtained according to Eq. 10, where $f_{x},f_{y},c_{x},c_{y}$ denote the camera parameters, and $d_{i}$ represents the depth value. In Table 5, we compare two models (10K_2D_6NN v.s. 10K_3D_6NN), where 6-nearest neighbours algorithm is utilized to construct graphs and $10,000$ points are sampled at the first scale. We can find 10K_3D_6NN slightly outperforms 10K_2D_6NN on the RMSE metric. It is mainly because propagation in the camera (3D) coordinate system can learn the scene’s geometric structure.

Secondly, we discuss the performance of the model under different numbers of nearest neighbours. By setting $k$ ($k$ nearest neighbours) to different values, i.e., $3,6,9$, we train three models, i.e., 10K_3D_3NN ($k=3$), 10K_3D_6NN ($k=6$), and 10K_3D_9NN ($k=9$), all of which propagate features in the camera coordinate system. As shown in Table 5, in comparison to 10K_3D_3NN and 10K_3D_6NN, 10K_3D_9NN causes a slight decrease in the performance, it might be because increasing the number of nearest neighbours encourages the model to see unrelated contexts.

Lastly, we study the number of sampled points. In specific, we sample $10,000$, $8,000$, and $12,000$ points at the first scale, respectively, and at the following scales, half of points are sampled from the last scale. From Table 5, we can observe that more or fewer points might degrade the performance on the RMSE metric.

In a nutshell, the selection of coordinate system, the number of nearest neighbours and sampled points might affect the performance, but in most settings, the model performs well.

Analysis of branch integration. In Section 3.5, we introduce two methods for the integration of the two branches. Here, we analyze their performances. As shown in Table 6, the comparison (RMSE: 786 v.s. 789) between Feature-Integration (abbr. FI) and End-Integration (abbr. EI) shows that integration at the feature level is more powerful than the end in learning the reliability of the two branches.

In addition, we also evaluate the performance of the two branches. Taking the end-integration as an example, we report the performance of EI/Depth fusing the RGB information into the depth, and EI/Image doing the opposite. Although the two branches yield close scores on all metrics, by learning confidence maps to fuse them together, a significant improvement on all metrics is obtained. To understand the two branches deeply, we provide a qualitative example in Figure 8. It can be seen that the depth branch is able to generate dense depth map with higher confidence in most locations, while the image branch performs better in capturing the boundary information. This result also further supports that the two modalities are complementary to each other.

5 Generalization Capabilities on Different Levels of Sparsity

To show the generalization capabilities of ACMNet on different levels of sparsity, we evaluate our approach and other three state-of-the-art methods with publicly available code, i.e., Certainty , Sparse2dense , and NConv-CNN , on KITTI selected validation set under different input densities. In specific, we first uniformly sub-sample the raw LiDAR depth by ratios of $0.8,0.6,0.4,0.2,0.1,0.05$, and $0.025$ to generate sparse depth maps with different densities, and then test pretrained models on the generated sparse depth maps. Note that, all the models are trained on KITTI training set under the original sparsity (sampling ratio of $1.0$) but not fine-tuned on the new sparse depth maps. Figure 9 shows that our approach performs better under all input densities in terms of both RMSE and MAE metrics, which demonstrates the impressive generalization capabilities of our approach under different levels of sparsity.

Conclusion

In this paper, we have developed an Adaptive Context-Aware Multi-Modal Network (ACMNet) to recover a dense depth map from sparse LiDAR data and dense RGB data. The critical issue in depth completion is how to exploit the observed spatial contexts from multi-modal data efficiently. To this end, we apply the co-attention guided graph propagation within multiple graphs constructed from observed pixels, which adaptively extracts multi-scale and multi-modal features and contributes to the feature enhancement for unobserved pixels. Furthermore, to fuse the multi-modal features in an effective way, we propose the symmetric gated fusion strategy, which has the capability of learning the heterogeneity of the two modalities. Finally, we implement our ACMNet, where a stack of CGPMs are employed in the encoder and SGFMs are used in the decoder. Benefiting from the two new modules, ACMNet is capable of generating high-quality dense depth maps. Our extensive experiments have demonstrated the effectiveness of the network as well as the network components.

References