COOPERNAUT: End-to-End Driving with Cooperative Perception for Networked Vehicles

Introduction

The widespread deployment of autonomous driving and advanced driver assistance systems is challenged by safety concerns. While deep learning has improved autonomy stacks with data-driven techniques , learning-based driving policies to date are still brittle, especially in the face of extreme situations and corner cases that one might encounter only a few times every million miles of driving . The lack of robustness of learning algorithms is exacerbated by the limited sensing capabilities of optical sensors on individual vehicles, such as stereo cameras and LiDAR, that are confined to line-of-sight sensing and unreliable in bad weather conditions. With the advent of new telecommunication technologies, such as 5G networks and vehicle-to-vehicle (V2V) communications, cooperative perception is becoming a promising paradigm that enables sensor information to be shared between vehicles (and roadside devices ) in real-time. The shared information can augment the field of view of individual vehicles and convey the intents and path plans of nearby vehicles, offering the potential to improve driving safety, particularly in accident-prone scenarios.

Ideally, learning autonomous driving policies with cooperative perception should take advantage of existing deep learning methods customized for ego perception by considering the combined sensory data from all vehicles as an augmented version of on-board sensing. In practice, the efficacy of cooperative perception hinges on what data to transmit within the limited network bandwidth and how to use the aggregated information to build a coherent and accurate understanding of traffic situations. Recent work on cooperative driving has demonstrated the benefit of cross-vehicle perception for augmenting sensing capabilities and driving decisions . Nonetheless, these methods have abstracted away raw sensory data with low-dimensional meta-data. Prior work introduced 3D sensor fusion (AVR , Cooper ) and representation fusion (V2VNet ) algorithms that aggregate perception results from nearby vehicles via V2V channels. They focused on 3D detection and motion forecasting on static datasets, rather than interactive driving policies.

We introduce Coopernaut, an end-to-end cooperative driving model for networked vehicles. Coopernaut learns to fuse encoded LiDAR information shared by nearby vehicles under realistic V2V channel capacity. To communicate meaningful scene information from nearby vehicles while conforming to bandwidth limits, we design our driving policy architecture based on the Point Transformer , a self-attention network for point cloud processing. This architecture pre-processes the raw point cloud, on each networked vehicle locally, into spatial-aware neural representations. These representations are compact, which can be efficiently transmitted over realistic wireless channels. Meanwhile, they are physically grounded, thus can be spatially transformed and aggregated with ego representations. The entire architecture is end-to-end differentiable, permitting control supervision (imitating an oracle planner with access to privileged information) to flow back to the perception stack, thus ensuring the learned representations and messages contain task-relevant information.

To examine the effectiveness of Coopernaut, we develop a CARLA-based simulation framework, AutoCastSim, where we designed three accident-prone scenarios. All the scenarios are designed to be challenging for ego perception to fully comprehend the traffic situation. AutoCastSim has a built-in networking simulation for customizable multi-vehicle communications and an expert driving model with privileged information. We evaluate Coopernaut with voxel-based baselines and different sensor fusion schemes.

In summary, our main contributions are as follows:

We introduce Coopernaut, an end-to-end driving model with cooperative perception via V2V channels. Our model learns compact representations for communication that can be easily harnessed by the ego vehicle to improve its driving decisions.

We develop a network-augmented autonomous driving simulation framework AutoCastSim to evaluate Coopernaut and baselines in accident-prone scenarios and to promote future research on vision-based cooperative perception.

Our results show that Coopernaut substantially reduces safety hazards for line-of-sight sensing. Its design improves both driving performance and communication efficiency over baselines.

Related Work

Learning a driving controller involves training closed-loop policies using deep networks, usually via imitation learning and/or reinforcement learning. Imitation learning for autonomous driving was pioneered by Pomerleau , and has since then been extended to urban and more complex scenarios . Very recently, reinforcement learning has also made progress in autonomous driving , showing potential to train better policies in complex situations . However, reinforcement learning is known to be more data-hungry and requires engineering a high-quality reward function. We follow the imitation learning paradigm but use an expert oracle with complete global information for training efficiency.

3D Perception for Autonomous Driving. 3D perception has become more popular in autonomous driving due to the decreasing cost of commoditized LiDAR sensors. Zhou and Tuzel pioneered using 3D object detection in autonomous driving, and since then, it has been further developed as better models, and more advanced techniques have been discovered. Very recently, Prakash et al. also explored end-to-end driving using point cloud data. Two families of 3D perception backbones have been widely adopted: voxel-based methods discretize points to voxels ; and point-based methods directly operate on coordinates . Coopernaut uses a transformer-based architecture with point-based representations , which preserves high spatial resolutions with discretization and requires lower bandwidths to transmit without compression needed by prior work .

Networked Vehicles and Cooperative Perception. Network connectivity offers a great potential for improving the safety and reliability of self-driving cars. Vehicles can now share surrounding information via Vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2X) channels using wireless technologies, such as Dedicated Short Range Communication (DSRC) and cellular-assisted V2X (C-V2X) . These V2V/V2X communication devices are increasingly deployed in current and upcoming vehicle models ). The academic community has built city-scale wireless research platforms (COSMOS ) and large connected vehicle testbeds (e.g., MCity , DRIVE C2X ), to explore the feasibility of cooperative vehicles and applications. Prior work proposed cooperative perception systems that broaden the vehicle’s visual horizon by sharing raw visual information with other nearby vehicles. Such systems can be scaled up to dense traffic scenarios leveraging edge servers or in an ad-hoc fashion . Recent work proposed multi-agent perception models to process sensor information and share compact representations within a local traffic network. In contrast, we focus on cooperative driving of networked vehicles with onboard visual data and realistic networking conditions, advancing towards real-world V2V settings.

Coopernaut

Our goal is to learn a closed-loop policy that controls an autonomous ego vehicle, which receives LiDAR observations $O_{t}^{(\text{ego})}$ at time $t$. Assume that there exist a variable number of $N_{t}$ neighboring vehicles in the range of V2V communications at time $t$, where $O^{(i)}_{t}$ is the raw 3D point cloud from the onboard LiDAR of the $i$-th vehicle. The cooperative driving policy for the ego vehicle is to find a policy $\pi(a_{t}|O_{t}^{(\text{ego})},O^{(1)}_{t},\ldots,O^{(N_{t})}_{t})$ that makes control decisions $a_{t}$ based on the joint observations of the ego vehicle and the $N_{t}$ neighboring vehicles. Here $\pi$ is parameterized by a deep neural network and trained end-to-end. In principle, we can transmit all cross-vehicle observations to the ego vehicle and process them as a whole. In practice, we have to take into account the networking bandwidth limit, which only allows for the message size orders of magnitude smaller. We thus first process the raw point clouds into compact representations, which can be transmitted through the V2V channels in real-time.

2 Background: Point Transformer

Our model’s backbone is the Point Transformer , a newly-developed neural network structure that learns compact point-based representations from 3D point clouds. It reasons about non-local interactions among points and produces permutation-invariant representations, making itself effective in aggregating multi-vehicle point clouds. Here we provide a brief review of Point Transformers.

We adopt the same design as Zhao et al. , which uses vector self-attention to construct the Point Transformer Layer. We also apply subtraction between features and append a position encoding function $\delta$ to both the attention vector $\gamma$ and the transformed features $\alpha$:

where $\theta$ is an MLP with two linear layers and one ReLU.

A Point Transformer block is shown in Figure 2, which integrates the self-attention layer, linear projections, and a residual connection. The input is a set of 3D points $p$ with a feature $x$ of each point. This block enables local information exchange among points, and produces new feature vectors for each point. The down-sampling block in Figure 2 is to reduce the cardinality of the point sets. We perform farthest point sampling to the input set to obtain a well-spread subset, and then use kNN graph and (local) max pooling in the neighborhood to further condense the information to smaller sets of points. The output is a subset of the original input points with new features.

3 Our Model

We use cross-vehicle perception to augment the sensing capabilities of the ego vehicle for it to make more informed decisions under challenging situations. The key challenge is to transmit sensory information efficiently through realistic V2V channels, understand the traffic situation from the aggregated information, and determine the reactive driving action in real-time.

Our Coopernaut model, illustrated in Figure 2, is composed of a Point Encoder for each neighboring V2V vehicle to encode its sensory data into compact messages, a Representation Aggregator to integrate the messages from neighboring cars with the ego perception, and a Control Module which translates the integrated representations to driving commands.

Point Encoder. To reduce communication burdens, every V2V vehicle processes its own LiDAR data locally and encodes the raw 3D point clouds into keypoints, each associated with a compact representation learned by the Point Transformer blocks. We construct the encoder with three Point Transformer blocks accompanied by two down-sampling blocks, both with a downsampling rate of $(1,4,4)$. The final cardinality of intermediate representations is $P/16$, where $P$ is the number of points in the raw point cloud. In our experiments, we preprocess 65,536 raw LiDAR points to 2,048 points via voxel pooling, i.e., representing the points in a voxel grid using their voxel centroid.

Representation Aggregator. Messages transmitted from other vehicles need to be fused and interpreted by the ego vehicle. The Representation Aggregator (RA) for cooperative perception is implemented as a voxel max-pooling operation and a point transformer block. RA first spatially transforms the keypoints in other vehicles’ coordinates to the ego vehicle’s frame using their relative poses. This operation assumes accurate vehicle localization (e.g., using HD maps). It then aggregates the incoming messages that are spatially close via max-pooling all the points located inside the same voxel grid cell. Finally, it fuses the multi-view perception information with another Point Transformer block. The two operations above preserve the permutation invariance with respect to the ordering of other vehicles and can handle a variable number of sharing vehicles. For bandwidth control, Coopernaut receives messages from three randomly chosen V2V vehicles in the vicinity.

Control Module. The control module is a fully-connected neural network designed to make control decisions based on the received messages. These control decisions include the throttle, brake, and steering, denoted as scalar $T,B,S$ respectively. These values output from the model are first clipped to their valid ranges (e.g., for throttle). To comply with the speed limit rules, we apply a PID speed controller to prevent speeding.

4 Policy Learning

We train our model to imitate the expert policy with privileged information using DAgger . To warm-start policy learning, we first train the model using behavior cloning.

Behavior Cloning. Behavior Cloning is designed to minimize the distribution gap between the training policy and the expert policy. The goal is to find an optimal policy $\hat{\pi}$ such that the loss w.r.t. the expert’s policy $\pi_{\text{expert}}$, under its induced distribution of states $S$ is minimized, i.e.,

where $\eta_{1},\eta_{2},\eta_{3}$ are the coefficients of the loss for each action. All three coefficients are set to $1$ in our experiments.

DAgger. Limitations of behavior cloning for autonomous driving have been discussed in Codevilla et al. . DAgger address the covariance shift issues via online training. The core idea is to let the student policy interact with the environment under the supervision of the expert and record the expert’s actions on the same states visited by the student. The training dataset is iteratively aggregated, using a mixture of the student’s and expert’s actions. The sampling policy $\pi_{i}$ for the $i$-th iteration follows:

where $\beta_{i}=\beta_{0}\times\beta_{i-1}$ are exponentially decreasing from the initial $\beta_{0}$, representing the probability that the expert’s action is executed at the $i$-th iteration.

5 Implementation Details

When more than three neighboring vehicles send messages, we randomly select messages from three of the vehicles. All the neighbors encode their processed point clouds locally by the 3-block Point Encoder and send the messages of size $128\times(128,3)$ and warp the coordinates to the ego frame. We aggregate the merged representations by another block of Point Transformer. After global max pooling, the features are concatenated with the ego speed feature before passing to the fully connected layer.

Our model has a 90ms latency on an NVIDIA GTX3090 GPU, where the point encoder takes 80ms. Our model training consists of two stages: behavior cloning and DAgger. We first train every scenario-specific model by behavior cloning, then the final policy of behavior cloning serves as an initial student policy for DAgger. We collect 4 new trajectories and append them to the Dagger dataset every 5 epochs using a sampling policy (see §3.4) with $\beta_{0}=0.8$ during the DAgger stage. For all data used for training, 25% of them are collected under accident-prone scenarios (with an occluded collider vehicle inserted) and 75% of them are normal driving trajectories. For more details, please refer to our supplementary materials and project website.

AutoCastSim

We present AutoCastSim, a simulation framework which offers network-augmented autonomous driving simulation on top of CARLA . This simulation framework allows custom designs of various traffic scenarios for training and evaluating cooperative driving models. The simulated vehicles can be configured with realistic wireless communications. It also provides a path planning-based oracle expert with access to privileged environment information.

We designed three challenging traffic scenarios, shown in Figure 3, in AutoCastSim as our evaluation benchmark. These scenarios are selected from the pre-crash typology of the US National Highway Traffic Safety Administration (NHTSA) , where limited line-of-sight sensing affects driving decisions:

* Overtaking. A truck blocks the way of a sedan in a two-way single lane road with a dashed yellow lane divider. The truck also impedes the sedan’s view of the opposite lane. The ego car has to overtake with a lane change maneuver.

* Left Turn. The ego car tries to turn left on a left-turn yield light but encounters another truck in the opposite left-turn lane, blocking its view of the opposite lanes and potential straight-driving vehicles.

* Red Light Violation. The ego car is crossing the intersection when another vehicle is rushing the red light. LiDAR fails to sense the other vehicle because of the lined-up vehicles waiting for the left turn.

2 V2V Communication

To simulate realistic wireless communication, we use real V2V wireless radios to profile wireless bandwidth capacity and packet loss rate due to channel diversity between mobile agents. Specifically, we use three iSmartways DSRC radios and three C-V2X radios , mounted on top of moving vehicles, to measure the maximum capacity of continuous wireless transmission in practice. Table 1 shows the tested throughput and packet loss. It also shows the throughput of WiFi (802.11n, ac) for context. Note that the 802.11 series is not designed for mobile scenarios. Table 1 shows that V2V bandwidth is two orders of magnitude smaller than the indoor wireless capacity. The extremely limited bandwidth, in practice, poses significant challenges for designing the representations for V2V communication. We use the Winner II wireless channel model in our simulator and use the measured C-V2X radio capacity and packet loss rate in the channel model. We refer to prior work for the design and implementations of the coordination, scheduling, and the network transport layer.

3 Oracle Expert

The expert has access to the privileged information of the traffic scenarios. The information includes the point cloud from the LiDARs of all neighboring vehicles and the positions and speeds of these neighboring vehicles and other traffic participants. The expert transforms all of the point clouds from neighboring cars to its ego perspective (which is impractical due to the wireless bandwidth limit mentioned above). The transformed point cloud is fused for downstream obstacle detection and planning. The expert policy leverages all information above to analyze and avoid possible collisions. The path planning algorithm uses an A* trajectory planner with pose and distance heuristics. The expert moves at a target speed of $20$km/h.

Experiments

We first discuss the evaluation method and the experiment setup and then give a brief overview of our baselines. Next we present the main quantitative evaluation results of our methods against baselines. Finally, we provide further analysis and visualization to understand the quality of our learned model.

Scenario Configuration. We generate traces from the three scenarios we implemented in AutoCastSim (§4.1) for training and evaluation. These scenarios can be programmatically re-configured with key parameters, notably the number of vehicles, vehicle spawning locations, and vehicle cruising speeds. Random combinations of these parameters are sampled to procedurally generate traces with traffic situations of varying complexity — in some cases the ego vehicle has to take emergency actions to avoid potential collisions, while in other cases, cruising along the default route can reach the destination.

Dataset. Specifically, for each scenario, we use the expert agent (§4.3) to generate an initial training set of 12 traces with randomized scenario configurations, followed by another randomly configured 84 traces for DAgger. In the evaluation, we systematically test each model on a spectrum of 27 randomly selected accident-prone environment configurations over three repeated runs, each using different random seeds for background traffic. For fair comparison, we use a fixed set of 27 test configurations to evaluate all models.

Metrics. We report three metrics, Success Rate, Collision Rate, and Success weighted by Completion Time:

Success Rate (SR). A successful completion of the scenario is defined as the ego agent reaching a designated target location in a permissible time without collision or prolonged stagnation. The success rate is defined as the percentage of successful completions among all evaluated traces.

Collision Rate (CR). Collision is the most common failure mode. Collision rate is defined as the percentage of evaluation traces where the ego vehicle collides with any entity, such as vehicles, buildings, etc.

Success weighted by Completion Time (SCT). SR reflects overall task success or failure. It does not differentiate the amount of time a driving agent needs to complete the traces. We introduce a third metric to weigh the success rate by the completion time ratio between the expert and the agent:

2 Baselines

We compare Coopernaut with non-V2V and V2V driving baselines. For fair comparison, we adopt the same neighbor selection process (§3.5) among V2V approaches:

* No V2V Sharing. The non-sharing baseline makes decisions solely based on the onboard LiDAR data and ego speed. The model shares the same data processing scheme for an individual vehicle and point encoder architecture as our final model.

* Early Fusion. The Early Fusion model assumes an unrealistic communication bandwidth, with which it can transmit and fuse the entire raw point cloud data from all neighboring vehicles. While this method is intractable in practice, it serves as a baseline to examine our point-based architecture’s effectiveness in representation learning. To fit this model in GPU memory, we limit the size of the fused input points to 4,096. Like the previous baseline, Early Fusion also uses a 3-block Point Transformer encoder.

* Voxel GNN. We adapt V2VNet , which is designed for 3D detection and motion forecasting, to learn end-to-end driving. Every vehicle processes its local point cloud onboard and shares a voxel representation with the ego vehicle for control. It uses a graph neural network (GNN) in the ego frame as the aggregator. The control actions are predicted from the GNN-fused representations.

For fair comparison, all baselines and proposed approaches are independently trained over three repeated runs with the same training parameters (§3.5). We report the average performance over the three runs on the same scenario configurations (§5.1).

3 Quantitative Results

This section presents the empirical evaluations of all the models in the three benchmarking scenarios.

Scenario Completion. Table 2 shows the performance comparisons in each of the three traffic scenarios. In all three scenarios, the No V2V Sharing model has performed poorly, with less than 50% success rate for each scenario and high collision rates. All three cooperative driving models, including Early Fusion, Voxel GNN, and Coopernaut, have achieved substantially higher SR and SCT scores and lower collision rates than the No V2V Sharing baseline. It indicates that the V2V communication provides critical information about the traffic situation over the ego vehicle’s line-of-sight sensing to make informed driving decisions. The Early Fusion method improves over the non-V2V baseline over 30% in average success rate. However, the Early Fusion baseline requires transmitting raw point clouds across vehicles, leading to an unrealistic bandwidth requirement of $60$Mbps (before data compression).

In contrast, pre-processing raw sensory data into representations has dramatically reduced the bandwidth requirements while improving driving performances. Both Voxel GNN and Coopernaut perform sensory fusion on the representation level. In comparison to the other cooperative driving models, Coopernaut outperforms both Early Fusion and Voxel GNN baselines for all three scenarios. We hypothesize that the point-based representation learning of Coopernaut makes it robust to localization errors compared with fusing raw points in Early Fusion. Furthermore, the explicit representation of point 3D locations and the point sampling module of Coopernaut retain a high spatial resolution of its intermediate representations in contrast to the voxel-based feature maps used by Voxel GNN.

Bandwidth Requirement. As shown in Table 2, sharing raw point cloud at the LiDAR scanning rate of 10fps would require a wireless bandwidth of 60Mbps, far beyond the achievable bandwidths in the current (DSRC) and future (C-V2X or LTE-direct) V2V communication technology (expected less than 10Mbps, see Table 1). V2VNet claims a bandwidth requirement of 25 Mbps with point cloud compression, which is also beyond what current V2V radios can support. In our design, both Voxel GNN and Coopernaut requires less than 6Mbps bandwidth, a 4$\times$ reduction of the communication data sizes of V2VNet without compression.

When developing the V2V models, we carefully explored the design space of the sharable representation size and its bandwidth requirement for both Voxel GNN and Coopernaut. For example, if Coopernaut were to share a 32$\times$32 representation, it only needs 0.9 Mbps. However, the coarse information is insufficient for the model to attain a good performance. We find that a 128$\times$128 point representation meets the bandwidth requirements (Table 1) without substantial performance degradation.

We further test Coopernaut under varied traffic densities in the most challenging Left Turn scenario. Figure 4 shows that our method generalizes to variable traffic densities, consistently outperforming the No V2V Sharing baseline. Qualitatively, we observe that No V2V Sharing drives slower in denser traffic, reacting better to emergency situations. In contrast, V2V methods do not improve much in denser traffic, as they tend to be impacted by the increased stochasticity of incoming messages from changing neighbors. Nonetheless, Coopernaut outperforms the baselines in all traffic densities with over $30\%$ higher success rates than No V2V Sharing.

Qualitative Visualizations. Figure 5 shows an example evaluation trajectory from Left Turn. The left-turning ego vehicle (grey) can proactively avoid collision by yielding to the opposite-going cars with Coopernaut. A common failure pattern of the non-sharing model is that it drives ahead to its target location regardless of any traffic violators or potential colliders due to the limited line-of-sight of its ego perception. The transmitted messages through V2V channels help our model resolve the ambiguity with cross-vehicle perception, leading to safer driving decisions in this accident-prone situation.

4 Limitations and Future Work

While our cooperative perception model conforms to realistic wireless bandwidth, we do not take into account practical networking issues, including transmission latency, networking protocols, and repetitive or lost packets. Nonetheless, Coopernaut is robust to packet loss to a certain extent ($5\%$ as configured in AutoCastSim). Its random neighbor selection also adds another layer to endure packet loss from individual transmitters.

Furthermore, highly accurate vehicle localization is assumed, which is used by Coopernaut to transform the point-based representations from neighboring vehicles to the ego vehicle, even though AutoCastSim simulates slight errors in the pose and height estimation of a vehicle. In reality, without a high definition map (HDMap), localization error can yield up to meter-level displacement. Using HDMap can significantly improve location and pose estimation, which is commonly adopted in both industry and academia .

For fair comparison, we use the same down-sampling scheme for all point-based baselines and our approach, which proves to be effective in our scenarios with moving vehicles and large obstacles. For smaller objects like pedestrians, adaptive sampling schemes based on semantic information is a promising direction for future work. We would also like to extend the model architecture of Coopernaut to better incorporate temporal information for improving driving performance.

Conclusion and Future Work

This work investigates vision-based driving using cooperative perception for networked vehicles in a newly designed simulation benchmark AutoCastSim. We introduce Coopernaut, an end-to-end driving policy that encodes, aggregates, and analyzes 3D LiDAR data from networked vehicles. The point encoder and representation aggregator of Coopernaut retain detailed spatial information and are robust to a varying number of communicating vehicles. Our empirical results show that our method improves the robustness of autonomous driving policies in risk-sensitive traffic scenarios.

This work has ample room for future extension. Our method relies on a hand-engineered oracle for imitation learning. It leaves open questions to investigate adaptive strategies of when to communicate, what to encode in messages, and how to drive cooperatively, ideally without the need of an algorithmic oracle.

This work has taken place in the Robot Perception and Learning Lab (RPL) and Learning Agents Research Group (LARG) at UT Austin. RPL research is supported in part by NSF (CNS-1955523, FRR-2145283), the MLL Research Award from the Machine Learning Laboratory at UT-Austin, and the Amazon Research Awards. LARG research is supported in part by NSF (CPS-1739964, IIS-1724157, FAIN-2019844), ONR (N00014-18-2243), ARO (W911NF-19-2-0333), DARPA, Lockheed Martin, GM, Bosch, and UT Austin’s Good Systems grand challenge. Peter Stone serves as the Executive Director of Sony AI America and receives financial compensation for this work. The terms of this arrangement have been reviewed and approved by the University of Texas at Austin in accordance with its policy on objectivity in research.

References