V2X-Sim: Multi-Agent Collaborative Perception Dataset and Benchmark for Autonomous Driving

I Introduction

Perception is a fundamental capability for autonomous vehicles, which allows them to represent, identify, and interpret sensory input for understanding the complex surroundings. In literature, single-vehicle perception has been intensively studied thanks to the well-established driving datasets , and researchers have proposed various algorithms to deal with different downstream tasks .

Despite recent advances in single-vehicle perception, the individual viewpoint often results in degraded perception in long-range or occluded areas. A promising solution to this problem is through vehicle-to-everything (V2X) , a cutting-edge communication technology that enables dialogue between a vehicle and other entities, including vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I). With the aid of V2X communication, we are able to upgrade single-vehicle perception to collaborative perception, which introduces more viewpoints to help autonomous vehicles see further, better and even see through occlusion, thereby fundamentally enhancing the capability of perception.

Collaborative perception naturally draws on communication and perception. Its development requires expertise from both communities. Recently, the communication community has made enormous efforts to promote such a study ; however, only a few works have been proposed from the perspective of perception . One major reason for this is the lack of well-designed and organized collaborative perception datasets. Due to the immaturity of V2X and the cost of simultaneously operating multiple autonomous vehicles, it is very expensive and laborious to build such a real dataset for research communities. Therefore, we synthesize a comprehensive and publicly available dataset, named as V2X-Sim, to advance the study of collaborative perception for V2X-communication-aided autonomous driving.

To generate V2X-Sim, we employ SUMO , a micro-traffic simulation, to produce numerically-realistic traffic flow, and CARLA , a widely-used open-source simulator for autonomous driving research, to retrieve well-synchronized sensor streams from multiple vehicles as well as the road-side unit (RSU). Meanwhile, multi-modality sensor streams of different entities are recorded to enable cross-modality perception. In addition, diverse annotations including bounding boxes, vehicle trajectories, and semantic labels are provided to facilitate various downstream tasks. To better serve multi-agent, multi-modality, and multi-task perception research for autonomous driving, we further provide a benchmark for three crucial perception tasks (collaborative detection, tracking, and segmentation) on the proposed dataset using the state-of-the-art collaboration strategies . In summary, our contributions are two-fold:

We propose V2X-Sim, a comprehensive V2X perception dataset for autonomous driving, to support multi-agent multi-modality multi-task perception research.

We create an open-source testbed for collaborative perception methods, and provide a benchmark on three tasks to encourage more research in this field.

II Related Work

Autonomous driving dataset. Since the pioneering dataset KITTI was released, the autonomous driving community has been trying to increase the dataset comprehensiveness in terms of driving scenarios, sensor modalities, and data annotations. Regarding driving scenarios, current datasets cover crowded urban scenes , adverse weather conditions , night scenes , and multiple cities to enrich the data distribution. As for sensor modalities, nuScenes collects data with Radar, RGB cameras, and LiDAR in a 360∘ viewpoint; WoodScape captures data with fisheye cameras. Regarding data annotations, semantic labels in both images and point clouds are provided to enable semantic segmentation; 2D/3D box trajectories are offered to facilitate tracking and prediction. In summary, most real datasets emphasize the data comprehensiveness in single-vehicle situations, but ignore the multi-vehicle scenarios.

V2X system and dataset. By sharing information with other vehicles or the RSU, V2X mitigates the shortcomings of single-vehicle perception and planning such as the limited sensing range and frequent occlusion . Previous research has developed an enhanced cooperative microscopic traffic model in V2X scenarios, and studied the effect of V2X in traffic disturbance scenarios. proposes a multi-modal cooperative perception system that provides see-through, lifted-seat, satellite and all-around views to drivers. More recently, COOPER investigates raw-data level collaborative perception to improve the detection capability for autonomous driving. V2VNet proposes intermediate-feature level collaboration to promote the vehicle’s perception and prediction capability. Several works use multiple infrastructure sensors to jointly perceive the environment and employ output-level collaboration with vehicle-to-infrastructure communication . As for the datasets, simulate the V2V scenarios with different frames from KITTI . Yet, these datasets are unrealistic multi-agent datasets for the measurements are not captured by different viewpoints. Some other works use a platoon strategy for data capture , but they are biased because the observations were highly correlated with each other. The most relevant work is V2V-Sim , which is based on a high-quality LiDAR simulator . Unfortunately, V2V-Sim does not include the V2I scenario and is not publicly available. Moreover, OPV2V and CODD only support the detection task in the V2V scenario. Existing collaborative perception datasets are summarized in Table I: V2X-SimThis work extends the LiDAR-based V2V data in our previous work with more modalities, scenarios and downstream tasks. is currently the most comprehensive one with multi-agent multi-modality sensory streams in both V2V and V2I scenarios, and can support various downstream tasks such as multi-agent collaborative detection, tracking, and semantic segmentation.

III V2X-Sim Dataset

V2X-Sim could enable more research on the collaboration strategy among vehicles to achieve a more robust perception. This could fundamentally benefit autonomous driving, intelligent transportation systems, smart cities, etc..

Multi-modality data is essential for robust perception. To ensure the comprehensiveness of our dataset, we equip each vehicle with a sensor suite composed of RGB cameras, LiDAR, GPS, IMU, and RSU with RGB cameras and LiDAR.

Sensor configuration. On both the vehicle and RSU, the camera and LiDAR cover 360∘ horizontally to enable full-view perception. Specifically, each vehicle carries six RGB cameras following the nuScenes configuration ; the RSU is equipped with four RGB cameras toward four directions at the crossroad. We employ depth camera, semantic segmentation camera, semantic LiDAR, and BEV semantic segmentation camera in CARLA to obtain the corresponding ground-truth for each RGB camera. Note that the BEV semantic segmentation camera is based on orthogonal projection while the ego-vehicle semantic segmentation camera uses perspective projection. Table 2 summarizes the detailed sensor specification.

Sensor layout and coordinate system. The overall sensor layout and coordinate system is shown in Fig. 2. The BEV semantic camera shares the same x, y position with LiDAR yet is placed higher to ensure a certain size of field of view. Note that we invert the y-axis in CARLA and use a right-hand coordinate system following nuScenes .

Diverse annotations. To assist downstream tasks including detection, tracking and semantic segmentation, we provide various annotations such as 3D bounding boxes, pixel-wise and point-wise semantic labels. Each box is defined by the location of its center in x, y, z coordinates, and its width, length, and height. In total, there are 23 categories such as the pedestrian, building, ground, etc. In addition, precise depth values are provided for depth estimation.

III-B CARLA-SUMO co-simulation

We consider it a realistic V2X scenario when multiple vehicles with their own routes are simultaneously located in the same intersection. Each intersection is also equipped with one RSU with sensing capability. Regarding the traffic simulation, there are several non-public simulators which can explicitly generate data tailored for collaborative perception such as scenes with occlusion, and sensor range limitations . Yet in this work, we use open-source CARLA-SUMO co-simulation for traffic flow simulation and data recording. Vehicles are spawned in CARLA via SUMO to roam around in the town with random routes. Hundreds of vehicles are spawned in different towns ($Town03$, $Town04$ and $Town05$ that have crossroads as well as T-junctions in both the crowded downtown and suburb highway). We record several log files in different towns. Afterwards, at different junctions, we read out 100 scenes from the log files. Each scene has a 20-second duration, and we choose $M(M=2,3,4,5)$ vehicles as well as one RSU in each scene as intelligent agents to share information. Example scenarios are shown in Fig. 3 (a).

III-C Downstream tasks

Our dataset not only supports single-agent perception tasks such as 3D object detection, tracking, image-/point-cloud-based semantic segmentation, depth estimation, but also enables collaborative perception such as collaborative 3D object detection, tracking, and collaborative BEV semantic segmentation in urban driving scenes. We provide a benchmark for collaborative perception algorithms.

III-D Dataset statistics

Annotation statistics. We provide statistics on the annotations and objects to highlight the inclusiveness and diversity of our dataset. In Fig. 3 (b) we analyze the size of the cars’ bounding boxes within a 70m range from ego vehicles in each frame. The great variation of car sizes indicates that our scenes contain a diverse set of car makes and models that well includes most of the common real-world vehicles. Figure 3 (c) shows the annotation count in each frame for vehicles within 0-30m, 30-50m, and 50-70m ranges from each ego vehicle. It suggests that our dataset features both crowded scenes (up to 100 annotations within 50-70m from the ego vehicle) and less crowded driving scenarios (as low as 10 annotations within 30m from the ego vehicle). Figure 3 (d) contains statistics on the number of LiDAR points per annotation for single-agent and multi-agent scenarios respectively. The number of total LiDAR points of each object annotation increases when there are more than one agents observing the same object. Specifically, for a single agent, there are 183.83 points in each annotation on average, but the number goes up to 875.59 points per annotation for multiple agents.

Scene features. We analyze the distance between each two ego vehicles for every frame, as shown in Fig. 3 (e). An overwhelming percentage of the ego vehicles are presented within 20-30 meters from each other, suggesting they are geographically closely connected. The speed of cars within 70m from ego vehicles are shown in Fig. 3 (f). Given the fact that our scenes are selected near intersections, we notice that a major fraction of vehicles are slower than 10 km/h. However, the maximum speed is as high as 90+ km/h. Figure 3 (g) shows the percentage of annotations observed by a certain number of ego vehicles, up to 5. Over 60% of the annotations are observed by at least two ego vehicles.

IV Collaborative Perception Benchmark

We benchmark three crucial perception tasks in autonomous driving within the collaboration setting: detection, tracking, and semantic segmentation. The three tasks have been extensively studied since they generate essential perception knowledge for autonomous vehicles to make safer decisions. For performance evaluation, we follow the same evaluation protocol of the three tasks in the single-agent scenario, except that we utilize the information shared by other agents while the single-agent perception do not have access to such information. We report the results in two scenarios: (1) V2V only, and (2) V2X (V2V + V2I).

Dataset format and split. Our V2X-Sim dataset follows the same storage format of nuScenes , an authoritative multi-modality single-agent autonomous driving dataset. nuScenes collected real-world single-agent driving data to promote the single-agent autonomous driving research; while we simulate multi-agent scenarios to facilitate the next-generation V2X-aided collaborative perception technology. Each scene in our dataset contains a 20 second traffic flow at a certain intersection of three CARLA towns, and the multi-modality multi-agent sensory streams are recorded at 5Hz, meaning each scene is composed of 100 frames. We generate 100 scenes with a total of 10,000 frames, and in each scene, 2-5 vehicles are selected as the collaboration agents. We use 8,000/1,000/1,000 frames for training/validation/testing respectively, and we ensure that there is no overlap in terms of the intersections across training/validation/testing set. Each frame has data sampled from multiple agents (vehicles and RSU). There are 37,200 samples in the training set, 5,000 samples in the validation set, and 5,000 samples in the test set. The split is shared across tasks.

Implementation details. Bird’s-eye-view (BEV) is a widely used and powerful representation in autonomous driving because it can describe the surrounding objects and overall context via a compact top-down 2D map . Therefore, BEV-based representation is adopted in all three tasks: we use a 3D voxel grid to represent the 3D world, employ binary representation, and assign each voxel a positive label if the voxel contains point cloud data. Since the generated 3D voxel grid can be considered as a pseudo-image whose height dimension is the channel dimension, we can perform the efficient 2D convolution instead of the heavy 3D convolution. Specifically, we crop the points located in the region of $\times\times$ meters for vehicles defined in the ego-vehicle Euclidean coordinate and $\times\times$ meters for RSU. The width and length of each voxel are 0.25 meter, and the height is 0.4 meter, meaning the generated BEV-based pseudo-image has a dimension of $256\times 256\times 13$ ($W\times L\times H$). Note that the models in all the tasks consume the 3D BEV map and generate perception results in 2D BEV.

Benchmark models. We aim to benchmark collaborative perception strategies rather than the well-studied individual perception methods. We consider early/intermediate/late/no collaboration models for the benchmark. The intermediate models, including DiscoNet , V2VNet , When2com , and Who2com , are based on the communication of the intermediate features of the neural network. The methods in our benchmark are as follows:

Lower-bound: The single-agent perception model without collaboration which processes a single-view point cloud is considered as the lower-bound.

Co-lower-bound: Collaborative lower-bound fuses the output from different single-agent perception models.

Upper-bound: The early collaboration model which transmits raw point cloud data is the upper-bound.

DiscoNet : DiscoNet uses a directed collaboration graph with matrix-valued edge weight to adaptively highlight the informative spatial regions and reject the noisy regions of the messages sent by the partners. After adaptive message fusion, the updated features will be transmitted to the output head for perception.

V2VNet : V2VNet uses a pose-aware graph neural network to propagate agents’ information, and employs a convolutional gated recurrent unit based module to aggregate other agents’ information. After several rounds of neural message passing, the updated features are fed into the output head to generate perception results.

When2com : When2com employs attention-based mechanism for communication group construction: the partners with satisfactory correlation scores will be selected as the collaborators. After the attention-score-based weighted fusion, the updated features will be fed into the output head for perceptions. The model with pose information is marked by $*$.

Who2com : Who2com shares a similar idea with When2com, yet it uses handshake mechanism to select the collaborator: the partner with the highest score will be selected as the collaborator. The model with pose information is marked by $*$.

We implement a 3D perception pipeline that can be integrated with all of the communication methods mentioned above. Since the source codes of V2VNet is not publicly available, we re-implement the V2VNet in PyTorch according to its pseudo-code. For when2com/who2com, we borrow their communication modules from its official code. All of the intermediate collaboration modules use the same architecture and conduct collaboration at the same intermediate feature layer. Moreover, all of the methods are trained with the same setting to ensure that the performance gain comes from the collaboration instead of irrelevant techniques.

Problem definition. As the most crucial perception task in autonomous driving, 3D object detection aims to recognize and localize the objects in 3D scenes given a single frame, with the following tracking, prediction and planning modules all heavily relying on the detections. The models consume voxelized point cloud and output BEV bounding boxes.

Backbone and evaluation. We use a classic anchor-based detector composed of a convolutional encoder, a convolutional decoder, and an output header for classification and regression . Regarding the loss function, we use the binary cross-entropy loss to supervise the box classification and the smooth $L_{1}$ loss to supervise the box regression, following . We employ the generic BEV detection evaluation metric: Average Precision (AP) at Intersection-over-Union (IoU) thresholds of 0.5 and 0.7. We target the vehicle detection and report the results on the test set.

Quantitative results. Table III demonstrates the quantitative comparisons on AP (@IoU = 0.5/0.7). We find that: (1) the upper-bound performs best amongst all methods, and it improves lower-bound significantly by 41.1$\%$ and 51.6$\%$ in terms of AP@0.5 and AP@0.7 in the scenario of V2V only, validating the necessity of collaboration; (2) V2V and V2I jointly can generally improve the perception over V2V only with more viewpoints, e.g., adding V2I can bring an improvement of 9.4$\%$ for upper-bound, and 5.5$\%$ for V2VNet in terms of AP@0.5; (3) among the intermediate models, DiscoNet achieves the best performance via the well-designed distilled collaboration graph, V2VNet achieves the second best performance via the powerful neural message passing, and When2com/Who2com only achieve comparable performance with lower-bound since the attention-mechanism is not suitable in point-cloud-based collaborative perception: the agents usually need complementary information rather than a highly similar one; (4) the late collaboration model (co-lower-bound) hurts the detection performance because of introducing extra false positives from other vehicles.

Qualitative results. The qualitative results are shown in Fig. 4. We see that the collaboration can fundamentally mitigate the problems of long-range perception and occlusion.

IV-B Collaborative multi-object tracking in BEV

Problem definition. Different from detection, multi-object tracking requires the generation of temporally consistent perception results. Multi-object tracking in BEV is to use bounding boxes, object categories, and object identities to track different objects within a temporal sequence.

Evaluation metrics. We mainly utilize HOTA (Higher Order Tracking Accuracy) to evaluate our BEV tracking performance. HOTA can evaluate detection, association, and localization performance via a single unified metric. In addition, the classic multi-object tracking accuracy (MOTA) and multi-object tracking precision (MOTP) are also employed. MOTA can measure detection errors and association errors. MOTP solely measures localization accuracy.

Baseline tracker. We implement SORT as our baseline tracker. Given the detection results, SORT will combine the Kalman Filter and Hungarian algorithm to achieve an accurate and efficient tracking performance.

Quantitative results. Quantitative comparisons of BEV tracking are shown in Table IV. Similar to BEV detection, upper-bound achieves the best performance in terms of MOTA and HOTA. Meanwhile, adding V2I can improve MOTA largely yet cannot help too much in MOTP. Co-lower-bound shows good performance in localization accuracy (MOTP). A more advanced tracker is required to exploit the collaboration for filling the performance gap.

IV-C Collaborative semantic segmentation in BEV

Problem definition. We aim to conduct semantic segmentation in BEV using only geometry point cloud. In the collaborative perception scenarios, there are measurements collected by multiple agents with distinct viewpoints. Therefore, there are more information in the scene, facilitating the semantic scene understanding.

Baseline segmentation method and evaluation metrics. We follow the backbone architecture as well as the loss function of U-Net in our baseline method. The input is a BEV-based voxelized point cloud, and the output is BEV semantic segmentation. We label and predict seven categories as listed in Table V, while the remaining is unlabeled. In our benchmark, we evaluate the segmentation performance using mean Intersection-over-Union (mIoU).

Quantitative results. As illustrated in Table V, we find that: (1) V2VNet, DiscoNet, and upper-bound achieve comparable performance in terms of terrain and road categories; (2) the attention-based methods (i.e., when2com, who2com) performs worse because the attention-based mechanisms try to find the collaboration partners with high correlation scores. Whereas, in 3D perception, the collaborators with complementary information should be prioritized during the collaboration. Such nonalignment can make it quite hard for the attention model to learn; (3) there is a large gap between lower-bound and upper-bound regarding the two safety-critical categories: vehicle ($45.93\%$ v.s. $64.09\%$) and pedestrian ($20.59\%$ v.s. $31.54\%$), proving the values of collaboration; (4) employing V2V and V2I jointly can generally enhance the vehicle segmentation over using V2V solely; (5) co-lower-bound performs better than the lower-bound.

Qualitative results. Figure 5 shows the semantics segmentation results. The results of upper-bound restore the semantic information with rich point cloud data. Intermediate-based collaboration strategies V2VNet and DiscoNet can achieve satisfactory performance yet When2com and Who2com hurt the performance compared to the lower-bound.

IV-D Discussions on pose noise and compression ratio

We further examine the robustness of different intermediate models against realistic pose noise (Gaussian noise with a mean of $0.05m-0.25m$ and a standard deviation of $0.02cm$), as shown in Fig. 6. We can see that the models perform comparably with or without pose noise in the training phase, and all the intermediate methods have shown stable performance against the pose noise. The reason is that the intermediate feature map has a relatively low spatial resolution (each grid in the feature map denotes a coverage of $2m\times 2m$), thus is not vulnerable to noisy pose. Meanwhile, we employ an $1\times 1$ autoencoder to further compress the feature channel of the transmitted feature map. We test the models with different compression ratios, and we found that the jointly learned $1\times 1$ autoencoder can even improve the performance a little bit, and most intermediate models achieve comparable performance at different levels of compression, as shown in Fig. 7.

V Conclusion

We propose V2X-Sim dataset based on CARLA-SUMO co-simulation, in order to enable multi-agent collaborative perception research in autonomous driving. Our dataset provides multi-agent multi-modality sensor streams captured by the vehicles and road-side unit (RSU) in realistic traffic flows. Diverse annotations are provided to support a variety of 3D perception tasks. In addition, we benchmark several state-of-the-art collaborative perception methods in collaborative BEV detection, tracking, and semantic segmentation tasks. Future works include the simulation of latency issues as well as the development of novel evaluation metrics in collaborative perception tasks. We believe our work can inspire many relevant research areas including but not limited to autonomous driving, computer vision, multi-robot system, communication engineering, and machine learning.

Acknowledgment. The authors would like to thank anonymous reviewers for their helpful suggestions, and NYU high performance computing (HPC) for the support.

References