GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Monocular 3D Object Detection

Introduction

$3$D object detection is one of the fundamental problems in computer vision, where the task is to infer $3$D information of the object. Its applications include augmented reality , robotics , medical surgery , and, more recently path planning and scene understanding in autonomous driving . Most of the $3$D object detectors are extensions of the $2$D object detector Faster R-CNN , which relies on the end-to-end learning idea to achieve State-of-the-Art (SoTA) object detection. Some of these methods have proposed changing architectures or losses . Others have tried incorporating confidence or temporal cues .

Almost all of them output a massive number of boxes for each object and, thus, rely on post-processing with a greedy clustering algorithm called Non-Maximal Suppression (NMS) during inference to reduce the number of false positives and increase performance. However, these works have largely overlooked NMS’s inclusion in training leading to an apparent mismatch between training and inference pipelines as the losses are applied on all boxes before NMS but not on final boxes after NMS (see Fig. 1(a)).We also find that $3$D object detection suffers a greater mismatch between classification and $3$D localization compared to that of $2$D localization, as discussed further in Sec. A3.2 of the supplementary and observed in . Hence, our focus is $3$D object detection.

Earlier attempts to include NMS in the training pipeline have been made for $2$D object detection where the improvements are less visible. Recent efforts to improve the correlation in $3$D object detection involve calculating or predicting the scores via likelihood estimation or enforcing the correlation explicitly . Although this improves the $3$D detection performance, improvements are limited as their training pipeline is not end to end in the absence of a differentiable NMS.

To address the mismatch between training and inference pipelines as well as the mismatch between classification and $3$D localization, we propose including the NMS in the training pipeline, which gives a useful gradient to the network so that it figures out which boxes are the best-localized in $3$D and, therefore, should be ranked higher (see Fig. 1(b)).

An ideal NMS for inclusion in the training pipeline should be not only differentiable but also parallelizable. Unfortunately, the inference-based classical NMS and Soft-NMS are greedy, set-based and, therefore, not parallelizable . To make the NMS parallelizable, we first formulate the classical NMS as matrix operation and then obtain a closed-form mathematical expression using elementary matrix operations such as matrix multiplication, matrix inversion, and clipping. We then replace the threshold pruning in the classical NMS with its softer version to get useful gradients. These two changes make the NMS GPU-friendly, and the gradients are backpropagated. We next group and mask the boxes in an unsupervised manner, which removes the matrix inversion and simplifies our proposed differentiable NMS expression further. We call this NMS as Grouped Mathematically Differentiable Non-Maximal Suppression (GrooMeD-NMS).

In summary, the main contributions of this work include:

This is the first work to propose and integrate a closed-form mathematically differentiable NMS for object detection, such that the network is trained end-to-end with a loss on the boxes after NMS.

We propose an unsupervised grouping and masking on the boxes to remove the matrix inversion in the closed-form NMS expression.

We achieve SoTA monocular $3$D object detection performance on the KITTI dataset performing comparably to monocular video-based methods.

Related Work

3D Object Detection. Recent success in $2$D object detection has inspired people to infer $3$D information from a single $2$D (monocular) image. However, the monocular problem is ill-posed due to the inherent scale/depth ambiguity . Hence, approaches use additional sensors such as LiDAR , stereo or radar . Although LiDAR depth estimations are accurate, LiDAR data is sparse and computationally expensive to process . Moreover, LiDARs are expensive and do not work well in severe weather .

Hence, there have been several works on monocular $3$D object detection. Earlier approaches use hand-crafted features, while the recent ones are all based on deep learning. Some of these methods have proposed changing architectures or losses . Others have tried incorporating confidence , augmentation , depth in convolution or temporal cues . Our work proposes to incorporate NMS in the training pipeline of monocular $3$D object detection.

Non-Maximal Suppression. NMS has been used to reduce false positives in edge detection , feature point detection , face detection , human detection as well as SoTA $2$D and $3$D detection . Modifications to NMS in $2$D detection , $2$D pedestrian detection , $2$D salient object detection and $3$D detection can be classified into three categories – inference NMS , optimization-based NMS and neural network based NMS .

The inference NMS changes the way the boxes are pruned in the final set of predictions. uses weighted averaging to update the $z$-coordinate after NMS. solves quadratic unconstrained binary optimization while and use point processes and MAP based inference respectively. and formulate NMS as a structured prediction task for isolated and all object instances respectively. The neural network NMS use a multi-layer network and message-passing to approximate NMS or to predict the NMS threshold adaptively . approximates the sub-gradients of the network without modelling NMS via a transitive relationship. Our work proposes a grouped closed-form mathematical approximation of the classical NMS and does not require multiple layers or message-passing. We detail these differences in Sec. 4.2.

Background

Let $\mathcal{B}\!=\!\{b_{i}\}_{i=1}^{n}$ denote the set of boxes or proposals $b_{i}$ from an image. Let ${\bf s}\!=\!\{s_{i}\}_{i=1}^{n}$ and ${\bf r}\!=\!\{r_{i}\}_{i=1}^{n}$ denote their scores (before NMS) and rescores (updated scores after NMS) respectively such that $r_{i},s_{i}\geq 0~{}\forall~{}i$. $\mathcal{D}$ denotes the subset of $\mathcal{B}$ after the NMS. Let $\mathbf{O}=[o_{ij}]$ denote the $n\times n$ matrix with $o_{ij}$ denoting the $2$D Intersection over Union (IoU${}_{2\text{D}}$) of $b_{i}$ and $b_{j}$. The pruning function $p$ decides how to rescore a set of boxes $\mathcal{B}$ based on IoU${}_{2\text{D}}$ overlaps of its neighbors, sometimes suppressing boxes entirely. In other words, $p(o_{i})=1$ denotes the box $b_{i}$ is suppressed while $p(o_{i})=0$ denotes $b_{i}$ is kept in $\mathcal{D}$. The NMS threshold $N_{t}$ is the threshold for which two boxes need in order for the non-maximum to be suppressed. The temperature $\tau$ controls the shape of the exponential and sigmoidal pruning functions $p$. $v$ thresholds the rescores in GrooMeD and Soft-NMS to decide if the box remains valid after NMS.

$\vee$ denotes the logical OR while $\left\lfloor x\right\rceil$ denotes clipping of $x$ in the range $$. Formally,

2 Classical and Soft-NMS

NMS is one of the building blocks in object detection whose high-level goal is to iteratively suppress boxes which have too much IoU with a nearby high-scoring box. We first give an overview of the classical and Soft-NMS , which are greedy and used in inference. Classical NMS uses the idea that the score of a box having a high IoU${}_{2\text{D}}$ overlap with any of the selected boxes should be suppressed to zero. That is, it uses a hard pruning $p$ without any temperature $\tau$. Soft-NMS makes this pruning soft via temperature $\tau$. Thus, classical and Soft-NMS only differ in the choice of $p$. We reproduce them in Alg. 1 using our notations.

GrooMeD-NMS

Classical NMS (Alg. 1) uses $\operatornamewithlimits{argmax}$ and greedily calculates the rescore $r_{i}$ of boxes $\mathcal{B}$ and, is thus not parallelizable or differentiable . We wish to find its smooth approximation in closed-form for including in the training pipeline.

Classical NMS uses the non-differentiable hard $\operatornamewithlimits{argmax}$ operation (Line $6$ of Alg. 1). We remove the $\operatornamewithlimits{argmax}$ by hard sorting the scores ${\bf s}$ and $\mathbf{O}$ in decreasing order (lines $2$-$3$ of Alg. 2). We also try making the sorting soft. Note that we require the permutation of ${\bf s}$ to sort $\mathbf{O}$. Most soft sorting methods apply the soft permutation to the same vector. Only two other methods can apply the soft permutation to another vector. Both methods use $\mathcal{O}\left(n^{2}\right)$ computations for soft sorting . We implement and find that is overly dependent on temperature $\tau$ to break out the ranks, and its gradients are too unreliable to train our model. Hence, we stick with the hard sorting of ${\bf s}$ and $\mathbf{O}$.

1.2 NMS as a Matrix Operation

The rescoring process of the classical NMS is greedy set-based and only considers overlaps with unsuppressed boxes. We first generalize this rescoring by accounting for the effect of all (suppressed and unsuppressed) boxes as

using the relaxation of logical OR $\bigvee$ operator as $\sum$ . See Sec. A1 of the supplementary material for an alternate explanation of (2). The presence of $r_{j}$ on the RHS of (2) prevents suppressed boxes from influencing other boxes hugely. When $p$ outputs discretely as $\{0,1\}$ as in classical NMS, scores $s_{i}$ are guaranteed to be suppressed to $r_{i}=0$ or left unchanged $r_{i}=s_{i}$ thereby implying $r_{i}\leq s_{i}~{}\forall~{}i$. We write the rescores ${\bf r}$ in a matrix formulation as

The above two equations are written compactly as

as the solution to (5) with ${\bf{I}}$ being the identity matrix. Intuitively, if the matrix inversion is considered division in (6) and the boxes have overlaps, the rescores are the scores divided by a number greater than one and are, therefore, lesser than scores. If the boxes do not overlap, the division is by one and rescores equal scores.

Note that the ${\bf{I}}+{\bf{P}}$ in (6) is a lower triangular matrix with ones on the principal diagonal. Hence, ${\bf{I}}+{\bf{P}}$ is always full rank and, therefore, always invertible.

1.3 Grouping

We next observe that the object detectors output multiple boxes for an object, and a good detector outputs boxes wherever it finds objects in the monocular image. Thus, we cluster the boxes in an image in an unsupervised manner based on IoU${}_{2\text{D}}$ overlaps to obtain the groups $\mathcal{G}$. Grouping thus mimics the grouping of the classical NMS, but does not rescore the boxes. As clustering limits interactions to intra-group interactions among the boxes, we write (6) as

This results in taking smaller matrix inverses in (7) than (6).

We use a simplistic grouping algorithm, i.e., we form a group $\mathcal{G}_{k}$ with boxes having high IoU${}_{2\text{D}}$ overlap with the top-ranked box, given that we sorted the scores. As the group size is limited by $\alpha$, we choose a minimum of $\alpha$ and the number of boxes in $\mathcal{G}_{k}$. We next delete all the boxes of this group and iterate until we run out of boxes. Also, grouping uses IoU${}_{2\text{D}}$ since we can achieve meaningful clustering in $2$D. We detail this unsupervised grouping in Alg. 3.

1.4 Masking

Classical NMS considers the IoU${}_{2\text{D}}$ of the top-scored box with other boxes. This consideration is equivalent to only keeping the column of $\mathbf{O}$ corresponding to the top box while assigning the rest of the columns to be zero. We implement this through masking of ${\bf{P}}_{\mathcal{G}_{k}}$. Let ${\bf{M}}_{\mathcal{G}_{k}}$ denote the binary mask corresponding to group $\mathcal{G}_{k}$. Then, entries in the binary matrix ${\bf{M}}_{\mathcal{G}_{k}}$ in the column corresponding to the top-scored box are $1$ and the rest are . Hence, only one of the columns in ${\bf{M}}_{\mathcal{G}_{k}}\!\odot~{}{\bf{P}}_{\mathcal{G}_{k}}$ is non-zero. Now, ${\bf{I}}_{\mathcal{G}_{k}}+{\bf{M}}_{\mathcal{G}_{k}}\!\odot{\bf{P}}_{\mathcal{G}_{k}}$ is a Frobenius matrix (Gaussian transformation) and we, therefore, invert this matrix by simply subtracting the second term . In other words, $({\bf{I}}_{\mathcal{G}_{k}}+{\bf{M}}_{\mathcal{G}_{k}}\!\odot{\bf{P}}_{\mathcal{G}_{k}})^{-1}={\bf{I}}_{\mathcal{G}_{k}}-{\bf{M}}_{\mathcal{G}_{k}}\!\odot{\bf{P}}_{\mathcal{G}_{k}}$. Hence, we simplify (7) further to get

Thus, masking allows to bypass the computationally expensive matrix inverse operation altogether.

We call the NMS based on (8) as Grouped Mathematically Differentiable Non-Maximal Suppression or GrooMeD-NMS. We summarize the complete GrooMeD-NMS in Alg. 2 and show its block-diagram in Fig. 1(c). GrooMeD-NMS in Fig. 1(c) provides two gradients - one through ${\bf s}$ and other through $\mathbf{O}$.

1.5 Pruning Function

As explained in Sec. 3.1, the pruning function $p$ decides whether to keep the box in the final set of predictions $\mathcal{D}$ or not based on IoU${}_{2\text{D}}$ overlaps, i.e., $p(o_{i})=1$ denotes the box $b_{i}$ is suppressed while $p(o_{i})=0$ denotes $b_{i}$ is kept in $\mathcal{D}$.

Classical NMS uses the threshold as the pruning function, which does not give useful gradients. Therefore, we considered three different functions for $p$: Linear, a temperature $(\tau)$-controlled Exponential, and Sigmoidal function.

Linear Linear pruning function is $p(o)=o$.

Sigmoidal Sigmoidal pruning function is $p(o)=\sigma\left(\frac{o-N_{t}}{\tau}\right)$ with $\sigma$ denoting the standard sigmoid. Sigmoidal function appears as the binary cross entropy relaxation of the subset selection problem .

We show these pruning functions in Fig. 2. The ablation studies (Sec. 5.4) show that choosing $p$ as Linear yields the simplest and the best GrooMeD-NMS.

2 Differences from Existing NMS

Although no differentiable NMS has been proposed for the monocular $3$D object detection, we compare our GrooMeD-NMS with the NMS proposed for $2$D object detection, $2$D pedestrian detection, $2$D salient object detection, and $3$D object detection in Tab. 1. No method described in Tab. 1 has a matrix-based closed-form mathematical expression of the NMS. Classical, Soft and Distance-NMS are used at the inference time, while GrooMeD-NMS is used during both training and inference. Distance-NMS updates the $z$-coordinate of the box after NMS as the weighted average of the $z$-coordinates of top-$\kappa$ boxes. QUBO-NMS , Point-NMS , and MAP-NMS are not used in end-to-end training. proposes a trainable Point-NMS. The Structured-SVM based NMS rely on structured SVM to obtain the rescores. Adaptive-NMS uses a separate neural network to predict the classical NMS threshold $N_{t}$. The trainable neural network based NMS (NN-NMS) use a separate neural network containing multiple layers and/or message-passing to approximate the NMS and do not use the pruning function. Unlike these methods, GrooMeD-NMS uses a single layer and does not require multiple layers or message passing. Our NMS is parallel up to group (denoted by $\mathcal{G}$). However, $\left|\mathcal{G}\right|$ is, in general, $<<\left|\mathcal{B}\right|$ in the NMS.

3 Target Assignment and Loss Function

Target Assignment. Our method consists of M3D-RPN and uses binning and self-balancing confidence . The boxes’ self-balancing confidence are used as scores ${\bf s}$, which pass through the GrooMeD-NMS layer to obtain the rescores ${\bf r}$. The rescores signal the network if the best box has not been selected for a particular object.

We extend the notion of the best $2$D box to $3$D. The best box has the highest product of IoU${}_{2\text{D}}$ and gIoU${}_{3\text{D}}$ with ground truth $g_{l}$. If the product is greater than a certain threshold $\beta$, it is assigned a positive label. Mathematically,

with q(b_{j},g_{l})=\text{IoU{}_{2\text{D}}}(b_{j},g_{l})~{}\left(\frac{1+\text{gIoU{}_{3\text{D}}}(b_{j},g_{l})}{2}\right). gIoU${}_{3\text{D}}$ is known to provide signal even for non-intersecting boxes , where the usual IoU${}_{3\text{D}}$ is always zero. Therefore, we use gIoU${}_{3\text{D}}$ instead of regular IoU${}_{3\text{D}}$ for figuring out the best box in $3$D as many $3$D boxes have a zero IoU${}_{3\text{D}}$ overlap with the ground truth. For calculating gIoU${}_{3\text{D}}$, we first calculate the volume $V$ and hull volume $V_{hull}$ of the $3$D boxes. $V_{hull}$ is the product of gIoU${}_{2\text{D}}$ in Birds Eye View (BEV), removing the rotations and hull of the $Y$ dimension. gIoU${}_{3\text{D}}$ is then given by

Loss Function. Generally the number of best boxes is less than the number of ground truths in an image, as there could be some ground truth boxes for which no box is predicted. The tiny number of best boxes introduces a far-heavier skew than the foreground-background classification. Thus, we use the modified AP-Loss as our loss after NMS since AP-Loss does not suffer from class imbalance .

Vanilla AP-Loss treats boxes of all images in a mini-batch equally, and the gradients are back-propagated through all the boxes. We remove this condition and rank boxes in an image-wise manner. In other words, if the best boxes are correctly ranked in one image and are not in the second, then the gradients only affect the boxes of the second image. We call this modification of AP-Loss as Imagewise AP-Loss. In other words,

where ${\bf r}^{(m)}$ and $\mathcal{B}^{(m)}$ denote the rescores and the boxes of the $m^{\text{th}}$ image in a mini-batch respectively. This is different from previous NMS approaches , which use classification losses. Our ablation studies (Sec. 5.4) show that the Imagewise AP-Loss is better suited to be used after NMS than the classification loss.

Our overall loss function is thus given by $\mathcal{L}=\mathcal{L}_{before}+\lambda\mathcal{L}_{after}$ where $\mathcal{L}_{before}$ denotes the losses before the NMS including classification, $2$D and $3$D regression as well as confidence losses, and $\mathcal{L}_{after}$ denotes the loss term after the NMS, which is the Imagewise AP-Loss with $\lambda$ being the weight. See Sec. A2 of the supplementary material for more details of the loss function.

Experiments

Our experiments use the most widely used KITTI autonomous driving dataset . We modify the publicly-available PyTorch code of Kinematic-3D . uses DenseNet-121 trained on ImageNet as the backbone and $n_{h}\!=\!1{,}024$ using $3$D-RPN settings of . As is a video-based method while GrooMeD-NMS is an image-based method, we use the best image model of henceforth called Kinematic (Image) as our baseline for a fair comparison. Kinematic (Image) is built on M3D-RPN and uses binning and self-balancing confidence.

Data Splits. There are three commonly used data splits of the KITTI dataset; we evaluate our method on all three.

Test Split: Official KITTI $3$D benchmark consists of $7{,}481$ training and $7{,}518$ testing images .

Val 1 Split: It partitions the $7{,}481$ training images into $3{,}712$ training and $3{,}769$ validation images .

Val 2 Split: It partitions the $7{,}481$ training images into $3{,}682$ training and $3{,}799$ validation images .

Training. Training is done in two phases - warmup and full . We initialize the model with the confidence prediction branch from warmup weights and finetune using the self-balancing loss and Imagewise AP-Loss after our GrooMeD-NMS. See Sec. A3.1 of the supplementary material for more training details. We keep the weight $\lambda$ at $0.05$. Unless otherwise stated, we use $p$ as the Linear function (this does not require $\tau$) with $\alpha=100$. $N_{t},v$ and $\beta$ are set to $0.4$ , $0.3$ and $0.3$ respectively.

Inference. We multiply the class and predicted confidence to get the box’s overall score in inference as in . See Sec. 5.2 for training and inference times.

Evaluation Metrics. KITTI uses AP${}_{3\text{D}|R_{40}}$ metric to evaluate object detection following . KITTI benchmark evaluates on three object categories: Easy, Moderate and Hard. It assigns each object to a category based on its occlusion, truncation, and height in the image space. The AP${}_{3\text{D}|R_{40}}$ performance on the Moderate category compares different models in the benchmark . We focus primarily on the Car class following .

Tab. 2 summarizes the results of $3$D object detection and BEV evaluation on KITTI Test Split. The results in Tab. 2 show that GrooMeD-NMS outperforms the baseline M3D-RPN by a significant margin and several other SoTA methods on both the tasks. GrooMeD-NMS also outperforms augmentation based approach MoVi-3D and depth-convolution based D4LCN . Despite being an image-based method, GrooMeD-NMS performs competitively to the video-based method Kinematic (Video) , outperforming it on the most-challenging Hard set.

2 KITTI Val 1 3D Object Detection

Results. Tab. 3 summarizes the results of $3$D object detection and BEV evaluation on KITTI Val 1 Split at two IoU${}_{3\text{D}}$ thresholds of $0.7$ and $0.5$ . The results in Tab. 3 show that GrooMeD-NMS outperforms the baseline of M3D-RPN and Kinematic (Image) by a significant margin. Interestingly, GrooMeD-NMS (an image-based method) also outperforms the video-based method Kinematic (Video) on most of the metrics. Thus, GrooMeD-NMS performs best on $6$ out of the $12$ cases ($3$ categories $\times~{}2$ tasks $\times~{}2$ thresholds) while second-best on all other cases. The performance is especially impressive since the biggest improvements are shown on the Moderate and Hard set, where objects are more distant and occluded.

AP${}_{3\text{D}}$ at different depths and IoU${}_{3\text{D}}$ thresholds. We next compare the AP${}_{3\text{D}}$ performance of GrooMeD-NMS and Kinematic (Image) on linear and log scale for objects at different depths of $$meters and IoU${}_{3\text{D}}$matching criteria of$0.3\!\relbar\joinrel\mathrel{\RHD}\!0.7$in Fig. 3 as in . Fig. 3 shows that GrooMeD-NMS outperforms the Kinematic (Image) at all depths and all IoU${}_{3\text{D}}$ thresholds.

Comparisons with other NMS. We compare with the classical NMS, Soft-NMS and Distance-NMS in Tab. 4. More detailed results are in Tab. 8 of the supplementary material. The results show that NMS inclusion in the training pipeline benefits the performance, unlike , which suggests otherwise. Training with GrooMeD-NMS helps because the network gets an additional signal through the GrooMeD-NMS layer whenever the best-localized box corresponding to an object is not selected. Interestingly, Tab. 4 also suggests that replacing GrooMeD-NMS with the classical NMS in inference does not affect the performance.

Score-IoU${}_{3\text{D}}$ Plot. We further correlate the scores with IoU${}_{3\text{D}}$ after NMS of our model with two baselines - M3D-RPN and Kinematic (Image) and also the Kinematic (Video) in Fig. 4. We obtain the best correlation of $0.345$ exceeding the correlations of M3D-RPN, Kinematic (Image) and, also Kinematic (Video). This proves that including NMS in the training pipeline is beneficial.

Training and Inference Times. We now compare the training and inference times of including GrooMeD-NMS in the pipeline. Warmup training phase takes about $13$ hours to train on a single $12$ GB GeForce GTX Titan-X GPU. Full training phase of Kinematic (Image) and GrooMeD-NMS takes about $8$ and $8.5$ hours respectively. The inference time per image using classical and GrooMeD-NMS is $0.12$ and $0.15$ ms respectively. Tab. 4 suggests that changing the NMS from GrooMeD to classical during inference does not alter the performance. Then, the inference time of our method is the same as $0.12$ ms.

3 KITTI Val 2 3D Object Detection

Tab. 5 summarizes the results of $3$D object detection and BEV evaluation on KITTI Val 2 Split at two IoU${}_{3\text{D}}$ thresholds of $0.7$ and $0.5$ . Again, we use M3D-RPN and Kinematic (Image) as our baselines. We evaluate the released model of M3D-RPN using the KITTI metric. does not report Val 2 results, so we retrain on Val 2 using their public code. The results in Tab. 5 show that GrooMeD-NMS performs best in all cases. This is again impressive because the improvements are shown on Moderate and Hard set, consistent with Tabs. 2 and 3.

4 Ablation Studies

Tab. 6 compares the modifications of our approach on KITTI Val 1 Cars. Unless stated otherwise, we stick with the experimental settings described in Sec. 5. Using a confidence head (Conf+No NMS) proves beneficial compared to the warmup model (No Conf+No NMS), which is consistent with the observations of . Further, GrooMeD-NMS on classification scores (denoted by No Conf + NMS) is detrimental as the classification scores are not suited for localization . Training the warmup model and then finetuning also works better than training without warmup as in since the warmup phase allows GrooMeD-NMS to carry meaningful grouping of the boxes.

As described in Sec. 4.1.5, in addition to Linear, we compare two other functions for pruning function $p$: Exponential and Sigmoidal. Both of them do not perform as well as the Linear $p$ possibly because they have vanishing gradients close to overlap of zero or one. Grouping and masking both help our model to reach a better minimum. As described in Sec. 4.3, Imagewise AP loss is better than the Vanilla AP loss since it treats boxes of two images differently. Imagewise AP also performs better than the binary cross-entropy (BCE) loss proposed in . Using the product of self-balancing confidence and classification scores instead of using them individually as the scores to the NMS in inference is better, consistent with . Class confidence performs worse since it does not have the localization information while the self-balancing confidence (Pred) gives the localization without considering whether the box belongs to foreground or background.

Conclusions

In this paper, we present and integrate GrooMeD-NMS – a novel Grouped Mathematically Differentiable NMS for monocular $3$D object detection, such that the network is trained end-to-end with a loss on the boxes after NMS. We first formulate NMS as a matrix operation and then do unsupervised grouping and masking of the boxes to obtain a simple closed-form expression of the NMS. GrooMeD-NMS addresses the mismatch between training and inference pipelines and, therefore, forces the network to select the best $3$D box in a differentiable manner. As a result, GrooMeD-NMS achieves state-of-the-art monocular $3$D object detection results on the KITTI benchmark dataset. Although our implementation demonstrates monocular $3$D object detection, GrooMeD-NMS is fairly generic for other object detection tasks. Future work includes applying this method to tasks such as LiDAR-based $3$D object detection and pedestrian detection.

References

GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Monocular 3D Object Detection Supplementary Material

Appendix A1 Detailed Explanation of NMS as a Matrix Operation

The rescoring process of the classical NMS is greedy set-based and calculates the rescore for a box $i$ (Line $10$ of Alg. 1) as

where $\mathbf{d}_{<i}$ is defined as the box indices sampled from $\mathbf{d}$ having higher scores than box $i$. For example, let us consider that $\mathbf{d}=\{1,5,7,9\}$. Then, for $i=7,~{}\mathbf{d}_{<i}=\{1,5\}$ while for $i=1,\mathbf{d}_{<i}=\phi$ with $\phi$ denoting the empty set. This is possible since we had sorted the scores ${\bf s}$ and $\mathbf{O}$ in decreasing order (Lines $2$-$3$ of Alg. 2) to remove the non-differentiable hard $\operatornamewithlimits{argmax}$ operation of the classical NMS (Line $6$ of Alg. 1).

Classical NMS only takes the overlap with unsuppressed boxes into account. Therefore, we generalize (15) by accounting for the effect of all (suppressed and unsuppressed) boxes as

The presence of $r_{j}$ on the RHS of (16) prevents suppressed boxes $r_{j}\approx 0$ from influencing other boxes hugely. Let us say we have a box $b_{2}$ with a high overlap with an unsuppressed box $b_{1}$. The classical NMS with a threshold pruning function assigns $r_{2}=0$ while (16) assigns $r_{2}$ a small non-zero value with a threshold pruning.

Although (16) keeps $r_{i}\geq 0$, getting a closed-form recursion in ${\bf r}$ is not easy because of the product operation. To get a closed-form recursion with addition/subtraction in ${\bf r}$, we first carry out the polynomial multiplication and then ignore the higher-order terms as

Dropping the $s_{i}$ in the second term of (17) helps us get a cleaner form of (22). Moreover, it does not change the nature of the NMS since the subtraction keeps the relation $r_{i}\leq s_{i}$ intact as $p(o_{ij})$ and $r_{j}$ are both between $$.

We can also reach (17) directly as follows. Classical NMS suppresses a box which has a high IoU${}_{2\text{D}}$ overlap with any of the unsuppressed boxes ($r_{j}\approx 1$) to zero. We consider any as a logical non-differentiable OR operation and use logical OR $\bigvee$ operator’s differentiable relaxation as $\sum$ . We next use this relaxation with the other expression ${\bf r}\leq{\bf s}$.

When a box shows overlap with more than two unsuppressed boxes, the term $\sum\limits_{j=1}^{i-1}p(o_{ij})r_{j}>1$ in (17) or when a box shows high overlap with one unsuppressed box, the term $s_{i}<p(o_{ij})r_{j}$. In both of these cases, $r_{i}<0$. So, we lower bound (17) with a $\max$ operation to ensure that $r_{i}\geq 0$. Thus,

We write the rescores ${\bf r}$ in a matrix formulation as

We next write the above two equations compactly as

However, for a differentiable NMS layer, we need to avoid the recursion. Therefore, we first solve (21) assuming the $\max$ operation is not present which gives us the solution ${\bf r}\approx\left({\bf{I}}+{\bf{P}}\right)^{-1}{\bf s}$. In general, this solution is not necessarily bounded between and $1$. Hence, we clip it explicitly to obtain the approximation

Appendix A2 Loss Functions

We now detail out the loss functions used for training. The losses on the boxes before NMS, $\mathcal{L}_{before}$, is given by

$b_{conf}$ is the predicted self-balancing confidence of each box $b$, while $b_{\theta_{a}}$ and $b_{\theta_{h}}$ are its orientation bins . $g$ denotes the ground-truth. $\lambda_{conf}$ is the rolling mean of most recent $\mathcal{L}_{3\text{D}}$ losses per mini-batch , while $\lambda_{a}$ denotes the weight of the orientation bins loss. CE and Smoooth-L1 denote the Cross Entropy and Smooth L1 loss respectively. Note that we apply $2$D and $3$D regression losses as well as the confidence losses only on the foreground boxes.

As explained in Sec. 4.3, the loss on the boxes after NMS, $\mathcal{L}_{after}$, is the Imagewise AP-Loss, which is given by

Let $\lambda$ be the weight of the $\mathcal{L}_{after}$ term. Then, our overall loss function is given by

We keep $\lambda_{a}=0.35$ following and $\lambda=0.05$. Clearly, all our losses and their weights are identical to except $\mathcal{L}_{Imagewise}$.

Appendix A3 Additional Experiments and Results

We now provide additional details and results evaluating our system’s performance.

Training images are augmented using random flipping with probability $0.5$ . Adam optimizer is used with batch size $2$, weight-decay $5\times 10^{-4}$ and gradient clipping of $1$ . Warmup starts with a learning rate $4\times 10^{-3}$ following a poly learning policy with power $0.9$ . Warmup and full training phases take $80k$ and $50k$ mini-batches respectively for Val 1 and Val 2 Splits while take $160k$ and $100k$ mini-batches for Test Split.

A3.2 KITTI Val 1 Oracle NMS Experiments

As discussed in Sec. 1, to understand the effects of an inference-only NMS on $2$D and $3$D object detection, we conduct a series of oracle experiments. We create an oracle NMS by taking the Val Car boxes of KITTI Val 1 Split from the baseline Kinematic (Image) model before NMS and replace their scores with their true IoU${}_{2\text{D}}$ or IoU${}_{3\text{D}}$ with the ground-truth, respectively. Note that this corresponds to the oracle because we do not know the ground-truth boxes during inference. We then pass the boxes with the oracle scores through the classical NMS and report the results in Tab. 7.

The results show that the AP${}_{3\text{D}}$ increases by a staggering $>60$ AP on Mod cars when we use oracle IoU${}_{3\text{D}}$ as the NMS score. On the other hand, we only see an increase in AP${}_{2\text{D}}$ by $\approx 11$ AP on Mod cars when we use oracle IoU${}_{2\text{D}}$ as the NMS score. Thus, the relative effect of using oracle IoU${}_{3\text{D}}$ NMS scores on $3$D detection is more significant than using oracle IoU${}_{2\text{D}}$ NMS scores on $2$D detection. In other words, the mismatch is greater between classification and $3$D localization compared to the mismatch between classification and $2$D localization.

A3.3 KITTI Val 1 3D Object Detection

Comparisons with other NMS. We compare our method with the other NMS—classical, Soft and Distance-NMS and report the detailed results in Tab. 8. We use the publicly released Soft-NMS code and Distance-NMS code from the respective authors. The Distance-NMS model uses the class confidence scores divided by the uncertainty in $z$ (the most erroneous dimension in $3$D localization ) of a box as the Distance-NMS input. Our model does not predict the uncertainty in $z$ of a box but predicts its self-balancing confidence (the $3$D localization score). Therefore, we use the class confidence scores multiplied by the self-balancing confidence as the Distance-NMS input.

The results in Tab. 8 show that NMS inclusion in the training pipeline benefits the performance, unlike , which suggests otherwise. Training with GrooMeD-NMS helps because the network gets an additional signal through the GrooMeD-NMS layer whenever the best-localized box corresponding to an object is not selected. Moreover, Tab. 8 suggests that we can replace GrooMeD-NMS with the classical NMS in inference as the performance is almost the same even at IoU${}_{3\text{D}}$$=0.5$.

How good is the classical NMS approximation? GrooMeD-NMS uses several approximations to arrive at the matrix solution (22). We now compare how good these approximations are with the classical NMS. Interestingly, Tab. 8 shows that GrooMeD-NMS is an excellent approximation to the classical NMS as the performance does not degrade after changing the NMS in inference.

A3.4 KITTI Val 1 Sensitivity Analysis

There are a few adjustable parameters for the GrooMeD-NMS, such as the NMS threshold $N_{t}$, valid box threshold $v$, the maximum group size $\alpha$, the weight $\lambda$ for the $\mathcal{L}_{after}$, and $\beta$. We carry out a sensitivity analysis to understand how these parameters affect performance and speed, and how sensitive the algorithm is to these parameters.

Sensitivity to NMS Threshold. We show the sensitivity to NMS threshold $N_{t}$ in Tab. 9. The results in Tab. 9 show that the optimal $N_{t}=0.4$. This is also the $N_{t}$ in .

Sensitivity to Valid Box Threshold. We next show the sensitivity to valid box threshold $v$ in Tab. 10. Our choice of $v=0.3$ performs close to the optimal choice.

Sensitivity to Maximum Group Size. Grouping has a parameter group size $(\alpha)$. We vary this parameter and report AP${}_{3\text{D}|R_{40}}$ and AP${}_{\text{BEV}|R_{40}}$ at two different IoU${}_{3\text{D}}$ thresholds on Moderate Cars of KITTI Val 1 Split in Fig. 5. We note that the best AP${}_{3\text{D}|R_{40}}$ performance is obtained at $\alpha=100$ and we, therefore, set $\alpha=100$ in our experiments.

Sensitivity to Loss Weight. We now show the sensitivity to loss weight $\lambda$ in Tab. 11. Our choice of $\lambda=0.05$ is the optimal value.

Sensitivity to Best Box Threshold. We now show the sensitivity to the best box threshold $\beta$ in Tab. 12. Our choice of $\beta=0.3$ is the optimal value.

Conclusion. Our method has minor sensitivity to $N_{t},\alpha,\lambda$ and $\beta$, which is common in object detection. Our method is not as sensitive to $v$ since it only decides a box’s validity. Our parameter choice is either at or close to the optimal. The inference speed is only affected by $\alpha$. Other parameters are used in training or do not affect inference speed.

A3.5 Qualitative Results

We next show some qualitative results of models trained on KITTI Val 1 Split in Fig. 6. We depict the predictions of GrooMeD-NMS in image view on the left and the predictions of GrooMeD-NMS, Kinematic (Image) , and ground truth in BEV on the right. In general, GrooMeD-NMS predictions are more closer to the ground truth than Kinematic (Image) .

A3.6 Demo Video of GrooMeD-NMS

We next include a short demo video of our GrooMeD-NMS model trained on KITTI Val 1 Split. We run our trained model independently on each frame of the three KITTI raw sequences - 2011_10_03_drive_0047, 2011_09_29_drive_0026 and 2011_09_26_drive_0009. None of the frames from these three raw sequences appear in the training set of KITTI Val 1 Split. We use the camera matrices available with the raw sequences but do not use any temporal information. Overlaid on each frame of the raw input videos, we plot the projected $3$D boxes of the predictions and also plot these $3$D boxes in the BEV. We set the frame rate of this demo at $10$ fps. The demo is also available in HD at https://www.youtube.com/watch?v=PWctKkyWrno. In the demo video, notice that the orientation of the boxes are stable despite not using any temporal information.

Acknowledgements

This research was partially sponsored by Ford Motor Company and the Army Research Office under Grant Number W911NF-18-1-0330. This document’s views and conclusions are those of the authors and do not represent the official policies, either expressed or implied, of the Army Research Office or the U.S. Government.

We thank Mathieu Blondel and Quentin Berthet from Google Brain, Paris, for several useful discussions on differentiable ranking and sorting. We also discussed the logical operators’ relaxation with Ashim Gupta from the University of Utah. Armin Parchami from Ford Motor Company suggested the learnable NMS paper . Enrique Corona and Marcos Paul Gerardo Castro from Ford Motor Company provided feedback during the development of this work. Shengjie Zhu from the Computer Vision Lab at Michigan State University proof-read our manuscript and suggested several changes. We also thank Xuepeng Shi from University College London for sharing the Distance-NMS code for bench-marking. We finally acknowledge anonymous reviewers for their feedback that helped in shaping the final manuscript.