Geometry Uncertainty Projection Network for Monocular 3D Object Detection

Introduction

3D object detection is an important component in autonomous driving and has received increasing attention in recent years. Compared with the LiDAR/stereo-based methods , monocular 3D object detection is still a challenging task due to the lack of depth cues, which makes monocular object-level depth estimation naturally ill-posed. Therefore, the monocular 3D detector cannot achieve satisfactory performance even some complex network structures are applied. Recently, to alleviate this problem, some works attempt to introduce geometry priors to help depth inference, of which a widely used prior is the perspective projection model.

Existing methods with the projection model usually estimate the height of 2D and 3D bounding box first and then infer the depth via the projection formula $depth=h_{3d}\cdot f/h_{2d}$ ($f$ is the camera focal length). Depth inferred by this formula is highly related to the estimated 2D/3D heights so the error of the height estimation will also be reflected at the estimated depth. However, the error of height estimation is inevitable especially for the ill-posed 3D height estimation (2D height estimation is relatively more accurate because of the well-developed 2d detection), so we are more concerned about the depth inference error caused by the 3D height estimation error. To show the influence of this property, we visualize the depth shifts caused by a fixed 3D height error in Figure 2. We can find that a slight bias (0.1m) of 3D heights could cause a significant shift (even 4m) in the projected depth. This error amplification effect makes outputs of the projection-based methods hardly controllable, significantly affecting both inference reliability and training efficiency. In this paper, we propose a Geometry Uncertainty Projection Network that includes a Geometry Uncertainty Projection (GUP) module and a Hierarchical Task Learning (HTL) strategy to treat these problems.

The first problem is inference reliability. A small quality change in the 3D height estimation would cause a large change in the depth estimation quality. This makes the model cannot predict reliable uncertainty or confidence easily, leading to uncontrollable outputs. To tackle this problem, the GUP module is proposed to infer the depth based on the distribution form rather than a discrete value (see Figure 1). The depth distribution is inferred by the estimated 3D height distribution. So, the statistical characteristics of the estimated 3D height estimation would be reflected in the output depth distribution, which leads to more accurate Uncertainty. At the inference, this well-learned uncertainty would be mapped to a confidence value to indicate the depth inference quality, which makes the total projection process more reliable.

Another problem is the instability of model training. In particular, at the beginning of the training phase, the estimation of 2D/3D height tends to be noisy, and the errors will be amplified and cause outrageous depth estimation. Consequently, the training process of the network will be misled, which will lead to the degradation of the final performance. To solve the instability of the training, we propose the Hierarchical Task Learning (HTL) strategy, aiming to ensure that each task is trained only when all pre-tasks (e.g. 3D height estimation is one of the pre-tasks of depth estimation) are trained well. To achieve that, the HTL first measures the learning situation of each task by a well-designed learning situation indicator. Then it adjusts weights for each loss term automatically by the learning situation of their pre-tasks, which can significantly improve the training stability, thereby boosting the final performance.

In summary, the key contributions of this paper are as follows:

We propose a Geometry Uncertainty Projection (GUP) module combining both mathematical priors and uncertainty modeling, which significantly reduces the uncontrollable effect caused by the error amplification at the inference.

For the training instability caused by task dependency in geometry-based methods, we propose a Hierarchical Task Learning (HTL) strategy, which can significantly improve the training efficiency.

Evaluation on the challenging KITTI dataset shows the overall proposed GUP Net achieves state-of-the-art performance around 20.11% and 14.72% on the car and the pedestrian 3D detection respectively on the KITTI testing set.

Related works

Monocular 3D object detection. The monocular 3D object detection aims to predict 3D bounding boxes from a single given image . Existing methods focus on deep representation learning and geometry priors . Deep3DBox firstly tried to solve the key angle prediction problem by geometry priors. DeepMANTA introduced the 3D CAD model to learn shape-based knowledge and guided to better dimension prediction results. GS3D utilized the ROI surface features to extract better object representations. M3DRPN gave a novel modified 3D anchor setting and proposed a depth-wise convolution to treat the monocular 3D detection task. MonoPair proposed a pair-wise relationship to improve the monocular 3D detection performance.

Except for these methods, many methods tried to introduce geometry projection to infer depth . Ivan et al. combined the keypoint method and the projection to do geometry reasoning. Decoupled3D used lengths of bounding box edges to project and get the inferred depth. Bao et al. combined the center voting with the geometry projection to achieve better 3D center reasoning. All of these projection-based mono3D methods did not consider the error amplification problem, leading to the limited performance.

Uncertainty based depth estimation. The uncertainty theory is widely used in the deep regression method , which can model both aleatoric and epistemic uncertainty. This technology is well developed in the depth estimation , which can significantly reduce the noise of the depth targets. However, these methods directly regressed the depth uncertainty by the deep models and neglected the relationships between the height and the depth. In this work, we try to compute the uncertainty via combining both end-to-end learning and the geometry relationships.

Multi-task learning. Multi-task learning is a widely studied topic in computer vision. Many works focus on task relation representation learning . Except that, some works also tried to adjust weights for different loss functions to solve the multi-task problem . GradNorm tried to solve the loss unbalance problem in joint multi-task learning and improved the training stability. Kendall et al. proposed a task-uncertainty strategy to treat the task balance problems, which also achieved good results. These loss weights control methods assumed that each task is independent from each other, which are unsuitable for our method, since the multiple tasks in our framework form a hierarchical structure, i.e., some tasks are dependent on their pre-tasks. Therefore, we propose a Hierarchical Task Learning strategy to handle it in this work.

Geometry Uncertainty Projection Network

Figure 3 shows the framework of the proposed Geometry Uncertainty Projection Network (GUP Net). It takes an image as input and processes it with a 2D detection backbone first, yielding 2D bounding boxes (Region of Interest, RoI) and then computes some basic 3D bounding box information,i.e., angle, dimensions and 3D projected center for each box. After that, the Geometry Uncertainty Projection (GUP) module predicts the depth distribution via combining both mathematical priors and uncertainty modeling. This depth distribution provides an accurate inferred depth value and its corresponding uncertainty. The predicted uncertainty would be mapped to 3D detection confidence at the inference stage. Furthermore, to avoid misleading caused by the error amplification at the beginning of training, an efficient Hierarchical Task Learning (HTL) strategy would control the overall training process, where each task does not start training until its pre-tasks have been trained well.

Our 2D detector is built on CenterNet , which includes a backbone network and three 2D detection sub-heads to compute the location, size, and confidence for each potential 2D box. As shown in Figure 3, the heatmap head computes a heatmap with the size of $(W\times H\times C)$ to indicate the coarse locations and confidences of the objects in the given image, where $C$ is the number of categories. Based on that, a 2D offset branch computes the bias $(\delta_{2d}^{u},\delta_{2d}^{v})$ to refine the coarse locations to the accurate bounding box center, and a 2D size branch predicts the size $(w_{2d},h_{2d})$ for each box. Their loss functions are denoted as $\mathcal{L}_{heatmap}$, $\mathcal{L}_{offset2d}$ and $\mathcal{L}_{size2d}$.

2 RoI feature representation

To guide the model to focus on the object, we crop and resize the RoI features using RoIAlign . The RoI features only contain the object-level features and do not include the background noise. However, those features lack location and size cues which are essential to the monocular depth estimation . Therefore, we compute the normalized coordinate map, and then concatenate it with the feature maps of each RoI in a channel-wise manner to compensate for that cues (shown as Figure 3).

3 Basic 3D detection Heads

With the extracted RoI features, we construct several sub-heads on top of those features to predict some basic 3D bounding box information. A 3D offset branch aims to estimate the 3D center projection on the 2D feature maps . The angle prediction branch predicts the relative alpha rotation angle . And the 3D size branch estimates the 3D dimension parameters, including height, width and length. These predictions are supervised by $\mathcal{L}_{offset3d}$, $\mathcal{L}_{angle}$ and $\mathcal{L}_{size3d}$, respectively. Note that $\mathcal{L}_{size3d}$ includes three parts for different dimensions, e.g., the height loss $\mathcal{L}_{h3d}$.

4 Geometry Uncertainty Projection

The basic 3D detection heads provide most information of the 3D bounding box except depth. Given the difficulty to regress depth directly, we propose a novel Geometry Uncertainty Projection model. The overall module builds the projection process in the probability framework rather than single values so that the model can compute the theoretical uncertainty for the inferred depth, which can indicate the depth inference reliability and also be helpful for the depth learning.

To achieve this goal, we first assume the prediction of the 3D height for each object is a Laplace distribution $La(\mu_{h},\lambda_{h})$ The probability density function of a Laplace random variable $X\sim La(\mu,\lambda)$ is: $f_{X}(x)=\frac{1}{2\lambda}\text{exp}(\frac{|x-\mu|}{\lambda})$, where $\mu$ and $\lambda$ are parameters of the Laplace distribution. The standard deviation $\sigma$ has a relationship with the parameters $\lambda$: $\sigma=\sqrt{2}\lambda$. . The distribution parameters $\mu_{h}$ and $\sigma_{h}$ are predicted by the 3D size stream in an end-to-end way. The $\mu_{h}$ denotes the regression target output and the $\sigma_{h}$ is the uncertainty of the inference. Consequently, the 3D height loss function can be defined as:

The minimization of $\mathcal{L}_{h3d}$ make $\mu_{h}$ and the ground-truth height $h^{gt}_{3d}$ as close as possible. Particularly, the difficult or noise-labeled samples usually incur large $\sigma_{3d}$, indicating the low prediction confidence. Based on the learned $h_{3d}$ distribution, the depth distribution of the projection output $La(\mu_{p},\lambda_{p})$ can be approximated as:

where $X$ is the standard Laplace distribution $La(0,1)$. In this sense, the mean and standard deviation of the projection depth $d_{p}$ are $\frac{f\cdot\mu_{h}}{h_{2d}}$ and $\frac{f\cdot\sigma_{h}}{h_{2d}}$, respectively. To obtain better predicted depth, we add a learned bias to modify the initial projection results. We also assume that the learned bias is a Laplace distribution $La(\mu_{b},\lambda_{b})$ and independent with the projection one. Accordingly, the final depth distribution can be written as:

We refer to the final uncertainty $\sigma_{d}$ as Geometry based Uncertainty (GeU). This uncertainty reflects both the projection uncertainty and the bias learning uncertainty. With this formula, a small uncertainty of $h_{3d}$ will be reflected in the GeU value. To optimize the final depth distribution, we apply the uncertainty regression loss:

Note that we also assume the depth distribution belong to Laplace distribution here for simplification. The overall loss would push the projection results close to the ground truth $d^{gt}$ and the gradient would affect the depth bias, the 2D height and the 3D height simultaneously. Besides, the uncertainty of 3D height and depth bias is also trained in the optimization process.

During inference, the reliability of depth prediction is critical for real-world applications. A reliable inference system is expected to feedback high confidence for a good estimation and low score for a bad one. As our well-designed GeU has capability of indicating the uncertainty of depth, we further map it to a value between 0$\sim$1 by an exponential function to indicate the depth Uncertainty-Confidence (UnC):

It can provide more accurate confidence for each projection depth. Thus we use this confidence as the conditional 3D bounding box scores $p_{3d|2d}$ in the testing. The final inference score can be computed as:

This score represents both the 2D detection confidence and the depth inference confidence, which can guide better reliability.

5 Hierarchical Task Learning

The GUP module mainly addresses the error amplification effect in the inference stage. Yet, this effect also damages the training procedure. Specifically, at the beginning of the training, the prediction of both $h_{2d}$ and $h_{3d}$ are far from accurate, which will mislead the overall training and damage the performance. To tackle this problem, we design a Hierarchical Task Learning (HTL) to control weights for each task at each epoch. The overall loss is:

where $\mathcal{T}$ is the task set. $t$ denotes the current epoch index and $\mathcal{L}_{i}$ means the $i$-th task loss function. $w_{i}(t)$ is the loss weight for the $i$-th task at the $t$-th epoch.

HTL is inspired by the motivation that each task should start training after its pre-task has been trained well. We split tasks into different stages as shown in Figure 4 and the loss weight $w_{i}(t)$ should be associated with all pre-tasks of the $i$-th task. The first stage is 2D detection, including heatmap, 2D offset, 2D size. Then, the second stage is the 3D heads containing angle, 3D offset and 3D size. All of these 3D tasks are built on the ROI features, so the tasks in 2D detection stage are their pre-tasks. Similarly, the final stage is the depth inference and its pre-tasks are the 3D size and all the tasks in 2D detection stage since depth prediction depends on the 3D height and 2D height. To train each task sufficiently, we aim to gradually increase the $w_{i}(t)$ from 0 to 1 as the training progresses. So we adopt the widely used polynomial time scheduling function in the curriculum learning topic as our weighted function, which is adapted as follows:

where $T$ is the total training epochs and the normalized time variable $\frac{t}{T}$ can automatically adjust the time scale. ${\alpha_{i}}(t)$ is an adjust parameter at the $t$-th epoch, corresponding to every pre-task of the $i$-th task. Figure 5 shows that ${\alpha_{i}}$ can change the trend of the time scheduler. The larger ${\alpha_{i}}$ is, the faster $w_{i}(\cdot)$ increases.

From the definition of the adjust parameter, it is natural to decide its value via the learning situation of every pre-task. If all pre-task have been well trained, the ${\alpha_{i}}$ is expected be large, otherwise it should be small. This is motivated by the observation that human usually learn advanced courses after finishing fundamental courses. Therefore, ${\alpha_{i}}(t)$ is defined as:

where $\mathcal{P}_{i}$ is the pre-task set for the $i$-th task. $ls_{j}$ means the learning situation indicator of $j$-th task, which is a value between 0$\sim$1. This formula means that $\alpha_{i}$ would get high values only when all pre-tasks have achieved high $ls$ (trained well). For the $ls_{j}$, inspired by , we design a scale-invariant factor to indicate the learning situation:

where $\mathcal{L}^{\prime}_{j}(\hat{t})$ is the derivative of the $\mathcal{L}_{j}(\cdot)$ at the $\hat{t}$-th epoch, which can indicate the local change trend of the loss function. The $\mathcal{DF}_{j}(t)$ computes the mean of derivatives in the recent $K$ epochs before the $t$-th epoch to reflect the mean change trend. If the $\mathcal{L}_{j}$ drops quickly in the recent $K$ epochs, the $\mathcal{DF}_{j}$ will get a large value. So the ${ls_{j}}$ formula means comparing the difference between the current trend $\mathcal{DF}_{j}(t)$ and the trend of the first $K$ epochs at the beginning of training $\mathcal{DF}_{j}(K)$ for the $j$-th task. If the current loss trend is similar to the beginning trend, the indicator will give a small value, which means that this task has not trained well. Conversely, if a task tends to converge, the ${ls_{j}}$ will be close to 1, meaning that the learning situation of this task is satisfied.

Based on the overall design, the loss weight of each term can reflect the learning situation of its pre-tasks dynamically, which can make the training more stable.

Experiments

Dataset. The KITTI 3D dataset is the most commonly used benchmark in the 3D object detection task, and it provides left camera images, calibration files, annotations for standard monocular 3D detection. It totally provides 7,481 frames for training and 7,518 frames for testing. Following , we split the training data into a training set (3,712 images) and a validation set (3,769 images). We conduct ablation studies based on this split and also report the final results with the model trained on all 7,481 images and tested by KITTI official server.

Evaluation protocols. All the experiments follow the standard evaluation protocol in the monocular 3D object detection and bird’s view (BEV) detection tasks. Following , we evaluate the ${\rm AP}_{40}$ to avoid the bias of original ${\rm AP}_{11}$.

Implementation details. We use DLA-34 as our backbone for both baseline and our method. The resolution of the input image is set to 380 × 1280 and the feature maps down-sampling rate is 4. Each 2D sub-head has two Conv layers (the channel of the first one is set to 256) and each 3D sub-head includes one 3x3 Conv layer with 256 channels, one averaged pooling layer and one fully-connected layer. The output channels of these heads are depending on the output data structure. We train our model with the batchsize of 32 on 3 Nvidia TiTan XP GPUs for 140 epochs. The initial learning rate is 1.25$e^{-3}$, which is decayed by 0.1 at the 90-th and the 120-th epoch. To make the training more stable, we apply the linear warm-up strategy in the first 5 epochs. The $K$ in the HTL is also set to 5.

2 Main Results

Results of Car category on the KITTI test set. As shown in Table 1, we first compare our method with other counterparts on the KITTI test set. Overall, the proposed method achieves superior results of Car category over previous methods, including those with extra data. Under fair conditions, our method achieves 3.74%, 3.19%, and 2.25% gains on the easy, moderate, and hard settings, respectively. Furthermore, our method also outperforms the methods with extra data. For instance, compared with the recently proposed CaDNN utilizing LiDAR signals as supervision of depth estimation sub-task, our method still obtains 0.94%, 0.79%, and 0.31% gains on the three difficulty settings, which confirms the effectiveness of the proposed method.

Results of Car category on the KITTI validation set. We also present our model’s performance on the KITTI validation set in Table 2 for better comparison, including different tasks and IoU thresholds. Specifically, our method gets almost the same performance as the best competing method MonoPair at the 0.5 IoU threshold. Moreover, our method improves with respect to MonoPair by 4.16%/4.77% for 3D/BEV detection under the moderate setting at 0.7 IoU threshold. This shows that our method is very suitable for high-precision tasks, which is a vital feature in the automatic driving scene. Note that RTM3D and RAR-Net do not report the ${\rm AP}_{40}$ metric on the validation set, and the comparison with them on ${\rm AP}_{11}$ metric can be found in supplementary materials.

Pedestrian/Cyclist detection on the KITTI test set. We also report the Pedestrian/Cyclist detection results in Table 1. Specifically, our method remarkably outperforms all the competing methods on all levels of difficulty for pedestrian detection. As for cyclist detection, our approach is superior to other methods except for MonoPSR and CaDNN. The main reason is those two methods can benefit from the extra depth supervision derived from LiDAR signals, thereby improving the overall performance. In contrast, the performances of the others are limited by the few training samples (there are 14,357/2,207/734 instances in total in the KITTI train-val set). It should be noted that our method still ranks first for methods without extra data.

Latency analysis. We also test the running time of our system. We test the averaged running time on a single Nvidia TiTan XP GPU and achieve 29.4 FPS, which shows the efficiency of the inference pipeline.

3 Ablation Study

To understand how much improvement each component provides, we perform ablation studies on the KITTI validation set for the Car category, and the main results are summarized in Table 3.

Effectiveness of the Coordinate Maps. We concatenate a Coordinate Map (CM) for each RoI feature, and the experiment (a$\rightarrow$b) clearly shows the effectiveness of this design, which means the location and size cues are crucial to our task. Note that the additional computing overhead introduced by CM is negligible.

Comparison of the Geometry Uncertainty Projection. We evaluate our Geometry Uncertainty Projection (GUP) module here. Note that, we think our GUP module brings gains from the following parts: geometry projection (GeP), Geometry based Uncertainty (GeU) and the Uncertainty-Confidence (UnC, Eq. 5). So we evaluate the effectiveness of these three parts respectively. First, we evaluate the effectiveness of the UnC. By comparing settings (b$\rightarrow$c and d$\rightarrow$e), we can find the UnC part can effectively and stably improve the overall performance, e.g. 1.09% improvement for (b$\rightarrow$c) and 0.78% improvement for (d$\rightarrow$e) on 3D detection task under moderate level. After that, we concern the GeP part effectiveness, we can see that adding GeP part improves the performance in the experiment (b$\rightarrow$d) without UnC, but leads to an accuracy drop in the experiment (c$\rightarrow$e) with UnC (The c and e experiments use directly learning uncertainty in the Eq. 5 to indicate confidence). This proves our motivation. It is hard for the projection-based model to directly learn accurate uncertainty and confidence because of the error amplification. Furthermore, note that the accuracy of hard cases decreases in both groups of experiments, which indicates the traditional projection cannot deal with the difficult cases caused by heavy occlusion/truncation. Second, we apply our GeU strategy based on GeP, and two groups of control experiments (e$\rightarrow$f and g$\rightarrow$h) are conducted. Comparing with c$\rightarrow$e, c$\rightarrow$f proves that our method can solve the difficulty of confidence learning in the projection-based model. The experimental results clearly demonstrate the effectiveness of our geometry modeling method for all metrics.

Influence of the Hierarchical Task Learning. We also quantify the contribution of the proposed Hierarchical Task Learning (HTL) strategy by two groups of control experiments (e$\rightarrow$g and f$\rightarrow$h), and both of them confirm the efficacy of the proposed HTL (improving performances for all metrics, and about 2% improvements for easy level). Also, we investigate the relationships between the loss terms and visualize the changing trend of loss weights in the training phase in Figure 7 to indicate the design effectiveness of our HTL scheme. It shows that the 2nd stage loss weights start increasing after all its tasks ({heatmap, 2D offset and 2D size}) close to convergence. And for the 3rd depth inference stage, it has a similar trend. Its loss weight starts increasing at about 11th epochs. At that time, all its pre-tasks {heatmap, 2D offset and 2D size, 3D size} have achieved certain progress.

To further prove that this strategy fits our method, we also compare our HTL with some widely used loss weight controllers in Table 4. We can see that our methods achieve the best performance. The main reason for the poor performance of the comparison methods is that our model is a hierarchical task structure. The task-independent assumption they request does not hold in our model. And for the GardNorm, its low performance is also caused by the error amplification effect. This effect makes the magnitude of the loss function significantly change in the total training phase so it is hard for the GardNorm to balance them.

4 Qualitative Results

For further investigating the effectiveness of our GUP Net. We show some bad cases and corresponding uncertainties from our model and the baseline projection method (the same setting in the 4th line in Table 3). The results are shown in Figure 6. We can see that our GUP Net can predict with high uncertainties for different bad cases including occlusion and far distance. And with the improvement of the prediction results, the uncertainty prediction of our method basically decreases. And the baseline projection model gives similar low uncertainty values for that bad case, which demonstrates the efficiency of our GUP Net.

Conclusion

In this paper, we proposed GUP Net model for the monocular 3D object detection to tackle the error amplification ignored by conventionally geometry projection models. It combines mathematical projection priors and the deep regression power together to compute more reliable uncertainty for each object, which not only be helpful for the uncertainty-based learning but also can be used to compute the accurate confidence in the testing stage. We also proposed a Hierarchical Task Learning strategy to learn the overall model better and reduce the instability caused by the error amplification. Extensive experiments validate the superior performance of the proposed algorithm, as well as the effectiveness of each component of the model.

Acknowledgement

This work was supported by the Australian Research Council Grant DP200103223, FT210100228, and Australian Medical Research Future Fund MRFAI000085.

References