RadarNet: Exploiting Radar for Robust Perception of Dynamic Objects

Introduction

Self-driving vehicles (SDVs) have to perceive the world around them in order to interact with the environment in a safe manner. Perception systems typically detect the objects of interest and track them over time in order to estimate their motion. Despite many decades of research, perception systems have not achieved the level of reliability required to deploy self-driving vehicles at scale without safety drivers.

Recent 3D perception systems typically exploit cameras , LiDAR , or their combination to achieve high-quality 3D object detection. While cameras capture rich appearance features, LiDAR provides direct and accurate 3D measurements. The sparsity of LiDAR measurements (e.g., at long range) and the sensor’s sensitivity to weather (e.g., fog, rain and snow) remain open challenges. In addition to detecting and recognizing objects, estimating their velocities is also of vital importance. In some safety critical situations, for example a child running out of occlusion in front of the SDV, the SDV needs to estimate velocities from a single measurement cycle in order to avoid collision. This estimation is often inaccurate (or even impossible) when using LiDAR or cameras alone as they provide static information only. While for pedestrians we may infer the motion from its pose with large uncertainty, for rigid objects like vehicles we can not make reasonable predictions from their appearance alone.

An appealing solution is to use sensors that are robust to various weather conditions and can provide velocity estimations from a single measurement. This is the case of Radar, which uses the Doppler effect to compute the radial velocities of objects relative to the SDV. Radar brings its own challenges, as the data is very sparse (typically much more so than LiDAR), the measurements are ambiguous in terms of position and velocity, the readings lack tangential information and often contain false positives. As a result, previous methods either focus on the ADAS by fusing Radar with cameras , where the performance requirements are relatively low; or fuse Radar data at the perception output level (e.g., tracks) , thus failing to fully exploit the complementary information of the sensors.

In this paper, we take a step forward in this direction and design a novel neural network architecture, dubbed RadarNet, which can exploit both LiDAR and Radar to provide accurate detections and velocity estimates for the actors in the scene. Towards this goal, we propose a multi-level fusion scheme that can fully exploit both geometric and dynamic information of Radar data. In particular, we first fuse Radar data with LiDAR point clouds via a novel voxel-based early fusion approach to leverage the Radar’s long sensing range. Furthermore, after we get object detections, we fuse Radar data again via an attention-based late fusion approach to leverage the Radar’s velocity readings. The proposed attention module captures the uncertainties in both detections and Radar measurements and plays an important role in transforming the 1D radial velocities from Radar to accurate 2D object velocity estimates.

We demonstrate the effectiveness of RadarNet on two large-scale driving datasets, where it surpasses the previous state-of-the-art in both 3D object detection and velocity estimation. We further show that exploiting Radar brings significant improvements in perceiving dynamic objects, improving both motion estimation and long range detection.

Related Work

As a high-quality 3D sensor, LiDAR has been widely used for 3D object detection in self-driving. Previous methods mainly differ in two aspects: the detection architecture and the input representation. While single-stage detectors have the advantages of simplicity and fast inference, two-stage methods are often superior in producing precisely localized bounding boxes. Different representations of LiDAR point clouds have been proposed: 3D voxel grids , range view (RV) projections , bird’s eye view (BEV) projections , and point sets are amongst the most popular. While 3D voxel grids are slow and wasteful to process due to the size of the volume which is mainly sparse, range view projections are dense representations by nature. However, RV images suffer from the large variance in object size and shape due to the projection. BEV projections achieve a better trade-off between accuracy and speed. Voxel features represented with either simple statistics or learned representations have been proposed. In this paper, we use a single-stage detector with BEV representation for its simplicity, effectiveness and efficiency.

0.2 Exploiting Radar for Perception:

Radar has long been used in ADAS for adaptive cruise control and collision avoidance due to its cost and robustness to severe weather conditions. Recently, Radar has been exploited in many other applications, spanning across free space estimation , object detection , object classification and segmentation . However, most of these methods treat Radar as another 3D sensor, ignoring its high-fidelity velocity information. In contrast, we exploit both Radar’s geometric and dynamic information thanks to a novel specialized fusion mechanism for each type of information.

0.3 Sensor Fusion with Radar:

In many self-driving perception systems, Radar data has been fused at the perception output level in the form of object tracks . Kalman Filter or IMM trackers are popular approaches to digest Radar data, and the resulting tracks are then fused with object tracks from other sensors. However, sensor fusion is not exploited during the process of generating those object tracks. Recent works also look at fusion between Radar and cameras within the perception system. Different Radar representations are proposed to facilitate fusion: spectrogram images , sparse locations in image space , pseudo-image by projecting to image space , BEV representation and object detections . However, these methods do not have high accuracy in terms of 3D perception. Instead, here we choose to fuse Radar with LiDAR and design a multi-level fusion mechanism that outperforms the state-of-the-art in self-driving.

Review of LiDAR and Radar Sensors

We first provide a review of LiDAR and Radar sensors and introduce our notation. We hope this short review can help readers better understand the intuitions behind our model designs, which will be described in the next section.

LiDAR (light detection and ranging) sensors can be divided into three main types: spinning LiDAR, solid state LiDAR, and flash LiDAR. In this paper we focus on the most common type: spinning LiDAR. This type of LiDAR emits and receives laser light pulses in and exploits the time of flight (ToF) to calculate the distance to the obstacles. As a result, LiDAR data is generated as a continuous stream of point clouds. We denote each LiDAR point as a vector $P=(x,y,z,t)$, encoding the 3D position and the capture timestamp. In practice we often divide the LiDAR data into consecutive sweeps for frame-wise point cloud processing. LiDAR is the preferred sensor for most self-driving vehicles due to its accurate 3D measurements. The main drawbacks are its sensitivity to dirt (which leads to poor performance in fog, rain and snow), cold (that causes exhaust plumes) as well as the lack of reflectivity of certain materials (such as windows and certain paints). Furthermore, its density decreases with range, making long range detection challenging.

Radar (radio detection and ranging) sensors work similarly as LiDAR, but transmit electromagnetic waves to sense the environment. The Radar outputs can be organized in three different levels: raw data in the form of time-frequency spectrograms, clusters from applying DBSCAN or CFAR on raw data, and tracks from performing object tracking on the clusters. From one representation to the next, the data sparsity and abstraction increases, while the noise in the data decreases. In this paper we focus on the mid-level data form, Radar clusters, for its good balance between information richness and noise. In the following we refer to these clusters as Radar targets. We denote each Radar target as a vector $Q=(\mathbf{q},v_{\|},m,t)$, where $\mathbf{q}=(x,y)$ is the 2D position in BEV, $v_{\|}$ is a scalar value representing the radial velocity, $m$ is a binary value indicating whether the target is moving or not, and $t$ is the capture timestamp. The main advantages of Radar are that it provides instantaneous velocity measurements and is robust to various weather conditions. However, its drawbacks are also significant. It has a low resolution and thus it is difficult to detect small objects. There are ambiguities (a modulo function) in range and velocity due to Radar aliasing, as well as false positive detections from clutter and multi-path returns. It is also worth noting that the objects’ real-world velocities (2D vectors in BEV) are ambiguous given only the radial velocity. Therefore we need to additionally estimate the tangential velocity or the 2D velocity direction in order to properly utilize the radial velocity.

We compare LiDAR and Radar data both quantitatively and qualitatively. We visualize both sensors’ data from the nuScenes dataset in Fig. 1, and we compare their technical specifications in Table 1. Note that LiDAR outperforms Radar in both accuracy and resolution by over an order of magnitude. The accurate 3D surface measurements makes LiDAR the first choice for high-precision 3D object detection. Radar can provide complementary information in two aspects: more observations at long range and instantaneous velocity evidence from the Doppler effect. We thus argue that since these sensors are very complementary, their combination provides a superior solution for self-driving.

Exploiting LiDAR and Radar for Robust Perception

In this section we present our novel approach to 3D perception, involving 3D object detection and velocity estimation. We refer the reader to Fig. 2 for an illustration of the overall architecture of our approach. To fully exploit the complementary information of the two sensor modalities and thereby benefit both object detection and velocity estimation, we propose two sensor fusion mechanisms, namely early fusion and late fusion, that operate at different granularities. More specifically, while early fusion learns joint representations from both sensor observations, late fusion refines object velocities via an attention-based association and aggregation mechanism between object detections and Radar targets.

1.2 Radar Voxel Representation:

Similar to how we accumulate multiple sweeps of LiDAR data, we also take multiple cycles of Radar data as input, in the same coordinate system as LiDAR. We keep only the $(x,y)$ position of Radar targets and ignore the height position as it is often inaccurate (if it ever exists). As a result, each cycle of Radar data can be voxelized as one BEV image. We concatenate multiple cycles along the channel dimension and use a motion-aware occupancy value as the feature for each voxel. Specifically, for each BEV voxel, if no Radar target falls into it, the voxel’s value is 0. If at least one moving Radar target (i.e., $m=1$) falls into it, the voxel’s value is 1. If all Radar targets falling into it are static, the voxel’s value is -1.

1.3 Early Fusion:

We use the same BEV voxel size for LiDAR and Radar data. Thus their voxel representations have the same size in BEV space. We perform early fusion by concatenating them together along the channel dimension.

2 Detection Network

We adopt a single-stage anchor-free BEV object detector with additional velocity estimation in the detection header.

We adopt the same backbone network architecture as PnPNet . The backbone network is composed of three initial convolution layers, three consecutive multi-scale inception blocks , and a feature pyramid network . The three initial convolution layers down-sample the voxel input by 4 and output 64-D feature maps. The inception block consists of three branches, each with a down-sampling ratio of $1\times$, $2\times$ and $4\times$ implemented by stride of the first convolution. The number of convolution layers in each branch is $2$, $4$ and $6$, and the number of feature channels in each branch is $32$, $64$ and $96$. The feature pyramid network merges multi-scale feature maps from the inception block into one, with $256$ channels for each layer. The final output of the backbone network is a $256$-D feature map with a $4\times$ down-sampling ratio compared to the voxel input.

2.2 Detection Header:

We apply a fully-convolutional detection header for anchor-free dense detection, which consists of a classification branch and a regression branch, each with $4$ convolution layers and $128$ channels. The detection is parameterized as $D=(c,x,y,w,l,\theta,\mathbf{v})$, which represents the confidence score, the object’s center position in BEV, its width, length and orientation, and its 2-D velocity $\mathbf{v}=(v_{x},v_{y})$ in BEV. The classification branch predicts the confidence score $c$, while the regression branch predicts all the other terms $(x-p_{x},y-p_{y},w,l,\cos(\theta),\sin(\theta),m,v_{x},v_{y})$, where $p_{x}$ and $p_{y}$ are the 2D coordinates of every voxel center and $m$ is an additional term that indicates the probability of moving. During inference, we set the 2-D velocity to $(0,0)$ if the predicted probability of moving is smaller than 50%.

3 Exploiting Dynamic Information via Late Fusion

While early fusion exploits the position and density information of Radar targets, late fusion is designed to explicitly exploit the Radar’s radial velocity evidence. Due to the lack of tangential information, the actual object velocity vector is ambiguous given the radial velocity alone. To address this issue, we propose to use the velocity estimation in object detections to align the radial velocity, which is simply back-projecting the radial velocity to the motion direction of the detection. We refer the reader to Fig. 3 for an illustration. It is thus apparent that the radial velocity is more confident when the angle between the radial direction and motion direction is small, as when it is close to , a very small variance in radial velocity will be exaggerated by back-projection.

Given a set of object detections and Radar targets, the key of fully exploiting Radar data lies in solving the following two tasks: (1) association of each Radar target with the correct object detection for velocity alignment; (2) aggregation to combine the velocity estimates from detection and associated Radar targets robustly. Both tasks are non-trivial to solve. The association is not a one-to-one mapping as there are many objects without any associated Radar targets, and there are also objects with multiple Radar targets. False positives and noisy positions of Radar targets also make association difficult. For the aggregation problem, it is hard to estimate the uncertainty of the Radar velocity as it also depends on the associated detection.

In this paper, we propose an attention-based mechanism that learns from data to both associate and aggregate. This is illustrated in Fig. 3. Specifically, given pairwise features defined between each object detection and Radar target, we first compute pairwise association scores via a learnable matching function. We then aggregate each detection with all Radar targets according to the normalized association scores to get the refined velocity estimate. Note that late fusion is performed on dynamic Radar targets only.

Given an object detection denoted as $D=(c,x,y,w,l,\theta,\mathbf{v})$ and a Radar target denoted as $Q=(\mathbf{q},v_{\|},m,t)$, we first define their pairwise feature as follows:

We then compute the pairwise association score by feeding the above feature to a learnable matching function:

In our case the matching function is parameterized as a Multi-Layer Perceptron (MLP) with five layers with 32, 64, 64, 64 and 1 channels respectively.

3.2 Velocity Aggregation:

We compute the association scores for all detections and Radar target pairs and refine the velocity estimate of each detection $D_{i}$ by aggregating information from all Radar targets. Towards this goal, we first normalize the association scores of all Radar targets to sum to 1. We append an additional score of $1$ before normalization to handle cases with no association.

We then refine the velocity magnitude by summing all the candidates (the detection itself as well as all Radar targets) weighted by their normalized scores:

The 2D velocity estimate is then computed as the refined velocity magnitude:

where the detection index $i$ is omitted for brevity.

4 Learning and Inference

We trained the proposed LiDAR and Radar fusion model with a multi-task loss defined as a weighted sum of the detection loss, velocity loss on the detection output, as well as the velocity loss on the late fusion output:

We use the Adam optimizer with batch normalization after every convolution layer and layer normalization after every fully-connected layer (except for the final output layer). For detection we use hard negative mining. $\mathcal{L}^{\text{det}}_{\text{reg}}$, $\mathcal{L}^{\text{velo}}_{\text{cls}}$ and $\mathcal{L}^{\text{velo}}_{\text{reg}}$ are computed on positive samples only, and $\mathcal{L}^{\text{velo}\_\text{attn}}_{\text{reg}}$ is computed on true positive detections only. We apply the same post-processing to generate final detections during training and testing phases, where the top 200 detections per class are kept and NMS is applied thereafter.

Experimental Evaluation

1.2 DenseRadar:

One advantage of Radar over LiDAR is its longer sensing range. To showcase this, we further evaluate our model on a self-collected dataset, called DenseRadar, with vehicle labels within 100m range for 5002 snippets. Velocity labels are estimated by fitting a kinematic bicycle model to the trajectory, which produces smoother velocities compared with the finite difference procedure employed in nuScenes. We use similar metrics as nuScenes. For detection we compute AP at 0.7 IoU in BEV. For velocity we report Average Dynamic Velocity Error (ADVE) on dynamic objects only. We make a training/validation split with 4666/336 logs each.

2 Implementation Details

We train a two-class model on nuScenes with a shared backbone network and class-specific detection headers. Global data augmentation is used during training, with random translations from m in the X and Y axes and [-0.2, 0.2]m in the Z axis, random scaling from [0.95, 1.05], random rotation from along the Z axis, and random left-right and front-back flipping. We do not apply augmentation at test time. To alleviate the class imbalance, we duplicate training frames that contain motorcycles by 5 times. The model is trained for 25 epochs with a batch size of 32 frames on 8 GPUs. We use an input voxel size of 0.125m in the X and Y axes, and 0.2m in the Z axis. We use $\alpha=1$ and $\beta=\delta=0.1$. Hyper-parameter tuning is conducted on the train-detect/train-track split.

We train a single-class model on DenseRadar. Since the dataset is much larger, we do not apply data augmentation. We use an input voxel resolution of 0.2m in all three axes due to the extra computation due to the longer detection range. We use $\alpha=1$ and $\beta=\delta=0.5$. The model is trained for 1.5 epochs.

3 Comparison with the State-of-the-Art

We compare our LiDAR and Radar fusion model with other state-of-the-art perception models on nuScenes and show the evaluation results in Table 2. Specifically, we compare with the camera-based method MonoDIS , the LiDAR-based methods PointPillar , PointPillar+ , 3DSSD , CBGS , and the LiDAR and camera fusion method PointPainting . RadarNet outperforms all methods significantly in both detection AP and velocity error. Compared with the second best on cars/motorcycles, our model shows an absolute gain of 2.2%/2.3% in detection AP and a relative reduction of 7%/21% in velocity error.

4 Ablation Study

We conduct an ablation study on the nuScenes and DenseRadar datasets to validate the effectiveness of our two-level fusion scheme. To better verify the advantage of the proposed attention-based late fusion, we build a strong baseline with carefully designed heuristics. Recall that our attention-based late fusion consists of two steps: association and aggregation. As a counterpart, we build the baseline fusion method by replacing each step with heuristics. In particular, for each detection candidate, we first use a set of rules to determine the Radar targets associated with it. Given a set of associated Radar targets (if any), we then take the median of their aligned velocities (by back-projecting to the motion direction of the detection) as the estimate from Radar and average it with the initial velocity estimate of the detection. If there are no associated Radar targets, we keep the original detection velocity.

Below we define the set of rules we designed for determining the associated Radar targets. Given the features in Eq. 2 and Eq. 3, a Radar target is considered as associated if it meets all of the following conditions:

We define these rules to filter out unreliable Radar targets, and the thresholds are chosen via cross-validation.

We show ablation results on nuScenes in Table 3. Note that our LiDAR only model already achieves state-of-the-art performance. Adding early fusion improves detection of motorcycles by 1.9% absolute AP, as the LiDAR observations are sparse and therefore Radar data serves as additional evidence. Early fusion does not affect the velocity performance much as only density information is exploited at present. When it comes to late fusion, our approach achieves over 14% velocity error reduction, significantly outperforming the heuristic baseline especially in motorcycles, where we typically have few Radar targets and therefore more noise.

4.2 Evaluation on DenseRadar:

Ablation results on DenseRadar are depicted in Table 4. We show detection APs in near range (0-40m), mid range (40-70m) and long range (70-100m) respectively. Early fusion helps long-distance object detection, bringing 0.8% absolute gain in the 70-100m range detection AP. When late fusion is added, larger improvements are achieved than on nuScenes (from 14% to 19%). Two reasons may account for this: (1) DenseRadar uses higher-end Radar sensors that produce denser returns; (2) we evaluate in longer range (100m vs. 50m), which is more challenging and therefore there is more room for improvement. However, the heuristic baseline still gets lower than 10% gain, showing the advantage of the proposed attention-based mechanism which can learn from noisy data.

5 Fine-Grained Analysis

To better understand in which aspects the velocity estimation performance is improved by exploiting Radar we conduct fine-grained evaluation on the larger-scale DenseRadar dataset with respect to different subsets of object labels. In particular, we create different subsets of labels by varying the object distance to the ego vehicle, number of observed LiDAR points, angle $\gamma$ between motion direction and radial direction, and the velocity magnitude.

We compare three model variants: LiDAR only, our model with heuristic late fusion and our model in Fig. 4. From the results we see that the heuristic model brings negligible gains when $\gamma>$$$or$\|\mathbf{v}\|<3$m/s. This justifies the and$1$ m/s thresholds in our heuristics as these are cases where Radar data contain large uncertainty. In contrast, our attention-based model consistently and significantly outperforms the heuristic model under all conditions, showing its effectiveness in capturing sensor uncertainties and exploiting both sensors.

6 Qualitative Results

In Fig. 5 we show the learned detection and Radar associations. Results are shown in sequence for each object to illustrate the temporal change in the association. From the results we observe that: (1) the association is sparse in that only relevant Radar targets are associated; (2) the association is quite robust to noisy locations of the Radar targets; (3) the model captures the uncertainty of Radar targets very well. For example, when the radial direction is near tangential to the object’s motion direction, the model tends to not associate any Radar targets as in such cases the Radar evidence is often very unreliable.

Conclusion

We have proposed a new method to exploit Radar in combination with LiDAR for robust perception of dynamic objects in self-driving. To exploit geometric information from Radar, we use a voxel-based early fusion approach, which is shown to improve long-distance object detection due to Radar’s longer sensing range. To exploit dynamic information, we propose an attention-based late fusion approach, which addresses the critical problem of associating Radar targets and objects without ground-truth association labels. By learning to associate and aggregate information, a significant performance boost in velocity estimation is observed under various conditions.

References