Source-linked AI summary
COOPERNAUT: End-to-End Driving with Cooperative Perception for Networked Vehicles
Jiaxun Cui, Hang Qiu, Dian Chen, Peter Stone, Yuke Zhu
TL;DR
Autonomous driving remains limited by brittle policies and vehicles’ line-of-sight sensing. COOPERNAUT uses end-to-end cooperative perception with compact LiDAR representations transmitted over realistic V2V channels, and its AutoCastSim evaluation reports improved success rates and lower bandwidth than baselines.
Problem
Autonomous driving reliability is limited by brittle data-driven policies in extreme situations and individual vehicles’ line-of-sight sensing.
Method
COOPERNAUT encodes, transmits, and aggregates compact, spatially grounded LiDAR representations from networked vehicles for end-to-end cooperative driving.
Results
COOPERNAUT improves autonomous-driving robustness in risk-sensitive traffic scenarios, with cooperative models achieving substantially higher success and lower collision rates than No V2V Sharing.
Takeaways & Limitations
Cooperative perception provides critical traffic information beyond ego vehicles’ line-of-sight sensing for more informed driving decisions in challenging scenarios.
Takeaways & Limitations
The evaluation does not account for transmission latency, networking protocols, or repetitive or lost packets, although the model is robust to packet loss to a certain extent.
Abstract
from arXiv · showhide
Optical sensors and learning algorithms for autonomous vehicles have dramatically advanced in the past few years. Nonetheless, the reliability of today's autonomous vehicles is hindered by the limited line-of-sight sensing capability and the brittleness of data-driven methods in handling extreme situations. With recent developments of telecommunication technologies, cooperative perception with vehicle-to-vehicle communications has become a promising paradigm to enhance autonomous driving in dangerous or emergency situations. We introduce COOPERNAUT, an end-to-end learning model that uses cross-vehicle perception for vision-based cooperative driving. Our model encodes LiDAR information into compact point-based representations that can be transmitted as messages between vehicles via realistic wireless channels. To evaluate our model, we develop AutoCastSim, a network-augmented driving simulation framework with example accident-prone scenarios. Our experiments on AutoCastSim suggest that our cooperative perception driving models lead to a 40% improvement in average success rate over egocentric driving models in these challenging driving situations and a 5 times smaller bandwidth requirement than prior work V2VNet. COOPERNAUT and AUTOCASTSIM are available at https://ut-austin-rpl.github.io/Coopernaut/.
1. Introduction
COOPERNAUT addresses brittle driving policies and limited line-of-sight sensing by sharing compact, spatially grounded LiDAR representations between vehicles. AUTOCASTSIM evaluates this approach in challenging accident-prone scenarios.
- Autonomous driving reliability is challenged by brittle learning-based policies in extreme situations and limited line-of-sight sensing from individual vehicles.
- Cooperative perception shares sensor information and nearby vehicles’ intents and path plans to augment an individual vehicle’s field of view.
- Prior cooperative-driving methods abstracted raw sensing into metadata or focused on 3D detection and motion forecasting rather than interactive driving.
- COOPERNAUT learns compact, physically grounded LiDAR representations that nearby vehicles transmit over realistic V2V channels and aggregate with ego representations.
- AUTOCASTSIM is a CARLA-based simulation framework with three accident-prone scenarios, customizable multi-vehicle networking, and an expert driving model.
- COOPERNAUT’s design improves driving performance and communication efficiency over baselines.
2. Related Work
Prior work studies learning-based driving and cooperative perception, but this paper focuses on networked vehicles using onboard visual data under realistic networking conditions. Its emphasis is cooperative driving in settings closer to real-world V2V operation.
- Deep learning driving controllers are commonly trained with imitation learning and, increasingly, reinforcement learning for complex situations.
- This work focuses on cooperative driving for networked vehicles with onboard visual data and realistic networking conditions, advancing toward real-world V2V settings.
3. COOPERNAUT
COOPERNAUT learns an end-to-end cooperative driving policy that compresses LiDAR observations into messages, aligns and aggregates them at the ego vehicle, and produces control decisions. Its training combines behavior cloning with DAgger to imitate an expert policy while accounting for bandwidth limits.
- 3.1. Problem Statement: COOPERNAUT’s policy makes ego-vehicle control decisions from joint LiDAR observations of the ego vehicle and neighboring vehicles.The policy is parameterized by a deep neural network and trained end-to-end.
- 3.2. Background: Point Transformer: Point Transformers produce compact, permutation-invariant point-based representations that support non-local interactions and multi-vehicle point-cloud aggregation.The architecture combines self-attention, linear projections, residual connections, and down-sampling through farthest-point sampling, kNN graphs, and local max pooling.
- 3.3. Our Model: 65,536 raw LiDAR points are preprocessed into 2,048 points via voxel pooling before the Point Encoder generates keypoints with learned features.The encoder uses three Point Transformer blocks and two downsampling blocks with downsampling rate (1, 4, 4), yielding intermediate cardinality P/16.
- 3.3. Our Model: The Representation Aggregator transforms neighboring keypoints into the ego frame, fuses messages with ego perception through voxel max-pooling, and passes joint representations to the Control Module.The aggregator assumes accurate vehicle localization, while the Control Module outputs throttle, brake, and steering decisions.
- 3.4. Policy Learning: The policy is trained in two stages: behavior cloning minimizes throttle, brake, and steering losses against an expert, followed by DAgger with iterative student-environment interaction.DAgger aggregates trajectories using a mixture of expert and student actions, with the expert-action probability decreasing exponentially across iterations.
4. AUTOCASTSIM
AUTOCASTSIM is a CARLA-based simulation framework for evaluating cooperative driving in challenging, accident-prone traffic scenarios with configurable wireless communications and privileged-information expert supervision.
- Framework: AUTOCASTSIM provides network-augmented autonomous-driving simulation on top of CARLA for training and evaluating cooperative driving models.Vehicles can be configured with realistic wireless communications, and the framework supports custom traffic scenarios.
- Scenarios: The benchmark includes three challenging scenarios selected from NHTSA’s pre-crash typology, where limited line-of-sight sensing affects driving decisions.The scenarios include overtaking and left-turn situations involving trucks that block views of relevant traffic.
- Scenarios: Figure 3 presents RGB BEV, ego-centric LiDAR BEV, and multi-vehicle fused LiDAR BEV views for each scenario.The figure also visualizes planned temporal trajectories; intersections at similar times indicate potential collisions.
- V2V Communication: AUTOCASTSIM models practical V2V communication using measured throughput and packet loss from off-the-shelf wireless radios.The simulator uses the Winner II wireless channel model with measured C-V2X capacity and packet loss.
- Expert: The privileged-information expert fuses neighboring vehicles’ point clouds after transforming them into its ego perspective.The expert also accesses neighboring and other traffic participants’ positions and speeds, although this point-cloud transformation is impractical under wireless bandwidth limits.
5. Experiments
Experiments evaluate COOPERNAUT and baselines in procedurally varied, accident-prone scenarios using success, collision, completion-time, and bandwidth metrics. Cooperative models outperform non-sharing driving, while COOPERNAUT combines stronger driving performance with lower communication requirements, though practical networking and localization assumptions remain.
- Evaluation Setup: 27 fixed test configurations were evaluated over three repeated runs with different background-traffic seeds.Training used 12 expert-generated traces followed by 84 DAgger traces.
- Metrics: Success Rate measures permissible target completion without collision or prolonged stagnation, while Collision Rate measures traces with any ego-vehicle collision.Success weighted by Completion Time additionally weights success by the expert-to-agent completion-time ratio.
- Scenario Completion: Cooperative models achieve higher success and completion-time scores and lower collision rates than No V2V Sharing across all three scenarios.The No V2V Sharing baseline remains below 50% success rate in each scenario.
- Scenario Completion: COOPERNAUT outperforms Early Fusion and Voxel GNN in all three scenarios.Its point-based representations explicitly preserve 3D locations and high spatial resolution, unlike Voxel GNN’s voxel-based feature maps.
- Traffic-Density Sensitivity: COOPERNAUT achieves over 30% higher success rates than No V2V Sharing across all tested traffic densities in Left Turn.Cooperative methods can be affected by increased stochasticity from changing neighbors in denser traffic.
- Qualitative Visualization: COOPERNAUT can avoid an occluded collision by using cross-vehicle perception to detect a red-light-violating vehicle.The qualitative Left Turn example contrasts this with non-sharing behavior that proceeds toward the target despite traffic violators or potential colliders.
- Limitations and Future Work: The evaluation omits transmission latency, networking protocols, repetitive packets, and packet loss, while assuming highly accurate vehicle localization.The simulation includes slight pose and height-estimation errors, and COOPERNAUT is reported as robust to 5% packet loss.
6. Conclusion and Future Work
This work introduces COOPERNAUT for cooperative perception driving and evaluates it in the AUTOCASTSIM simulation benchmark. The method improves robustness in risk-sensitive traffic scenarios, while future work remains necessary on oracle-free cooperation and adaptive communication.
- COOPERNAUT encodes, aggregates, and analyzes 3D LiDAR data from networked vehicles for end-to-end driving.Its point encoder and representation aggregator retain spatial information and handle varying numbers of communicating vehicles.
- AUTOCASTSIM is a newly designed simulation benchmark for evaluating vision-based cooperative perception in networked vehicles.
- COOPERNAUT improves the robustness of autonomous driving policies in risk-sensitive traffic scenarios.
- The method relies on a hand-engineered oracle for imitation learning, leaving oracle-free cooperative driving as an open direction.
- Future work includes adaptive decisions about when to communicate and what to encode in messages.