Source-linked AI summary
Collaboration Helps Camera Overtake LiDAR in 3D Detection
Yue Hu, Yifan Lu, Runsheng Xu, Weidi Xie, Siheng Chen, Yanfeng Wang
TL;DR
Camera-only 3D detection is economical but remains substantially worse than LiDAR-based detection, largely because cameras lack direct 3D measurements for precise depth estimation. CoCa3D addresses this gap through communication-efficient multi-agent collaboration, and improves prior state-of-the-art performance across three datasets, with cameras overtaking LiDAR-based detectors on OPV2V+ using 10 collaborative agents.
Problem
Camera-only 3D detection offers a cheaper, simpler alternative to LiDAR but is consistently worse in most scenarios because cameras lack direct 3D measurements for precise depth estimation.
Method
CoCa3D uses multi-agent collaboration to share sparse, informative depth and detection messages that disambiguate depth and complement occluded or long-range regions.
Results
CoCa3D improves previous state-of-the-art AP@70 performance by 44.21% on DAIR-V2X, 30.60% on OPV2V+, and 12.59% on CoPerception-UAVs+, while overtaking LiDAR-based detectors on OPV2V+ with 10 collaborative agents.
Takeaways & Limitations
With sufficient collaboration, camera-only detectors can achieve a stronger performance-bandwidth trade-off and potentially surpass LiDAR-based detectors in some practical scenarios.
Takeaways & Limitations
Real-world evaluation is constrained because DAIR-V2X is the sole public multi-agent perception dataset and contains only one vehicle and one roadside unit, so the work mainly relies on simulation data.
Abstract
from arXiv · showhide
Camera-only 3D detection provides an economical solution with a simple configuration for localizing objects in 3D space compared to LiDAR-based detection systems. However, a major challenge lies in precise depth estimation due to the lack of direct 3D measurements in the input. Many previous methods attempt to improve depth estimation through network designs, e.g., deformable layers and larger receptive fields. This work proposes an orthogonal direction, improving the camera-only 3D detection by introducing multi-agent collaborations. Our proposed collaborative camera-only 3D detection (CoCa3D) enables agents to share complementary information with each other through communication. Meanwhile, we optimize communication efficiency by selecting the most informative cues. The shared messages from multiple viewpoints disambiguate the single-agent estimated depth and complement the occluded and long-range regions in the single-agent view. We evaluate CoCa3D in one real-world dataset and two new simulation datasets. Results show that CoCa3D improves previous SOTA performances by 44.21% on DAIR-V2X, 30.60% on OPV2V+, 12.59% on CoPerception-UAVs+ for AP@70. Our preliminary results show a potential that with sufficient collaboration, the camera might overtake LiDAR in some practical scenarios. We released the dataset and code at https://siheng-chen.github.io/dataset/CoPerception+ and https://github.com/MediaBrain-SJTU/CoCa3D.
1. Introduction
Camera-only 3D detection offers a cheaper alternative to LiDAR but remains limited by depth ambiguity, occlusion, and long-range regions. CoCa3D addresses these issues through communication-efficient multi-agent collaboration, improving benchmark performance and narrowing the camera–LiDAR gap.
- 1. Introduction: Camera-only detection is more economical than LiDAR but is significantly and consistently worse in most scenarios.LiDAR provides precise 3D measurements, whereas cameras lack direct 3D measurements.
- 1. Introduction: CoCa3D uses multi-agent collaboration to share complementary information across viewpoints and improve camera-only 3D detection.The framework includes single-agent detection, collaborative depth estimation, and collaborative detection feature learning.
- 1. Introduction: 44.21% on DAIR-V2X, 30.60% on OPV2V+, and 12.59% on CoPerception-UAVs+ for AP@70 improve previous SOTA performance.The evaluation covers one real-world dataset and two simulation datasets.
- 1. Introduction: With sufficient collaboration, cameras can overtake LiDAR-based detectors on OPV2V+ and achieve a strong performance–bandwidth trade-off.The expanded datasets include cars, infrastructure, and drones as collaborating agents.
- 1. Introduction: Shared depth and detection messages address depth ambiguity, occlusion, and long-range limitations while communication selects spatially sparse critical cues.The approach targets more accurate and complete 3D representations.
2. Related Work
Camera-only 3D detection estimates 3D objects from 2D images, while LiDAR detection benefits from precise 3D measurements. Camera methods commonly build efficient BEV representations through depth-based projection or query-based cross-attention.
- 2. Related Work: Camera-only 3D detection detects objects in 3D space from 2D images through explicit or implicit depth estimation.The single-agent detector in this work follows the CaDDN depth-based approach.
- 2. Related Work: BEV representations are widely used because they offer computation efficiency and performance comparable to 3D voxel features.Depth-based methods estimate distributions before projecting image features, whereas query-based methods use BEV queries and cross-attention.
- 2. Related Work: LiDAR-based 3D detection achieves excellent performance because its inputs provide precise 3D measurements.Common encoders are voxel-based or point-based.
3. Collaborative Camera-Only 3D Detection
CoCa3D combines single-agent camera-only 3D detection with collaborative depth estimation and detection-feature learning. It shares spatially sparse depth and detection cues across agents to improve depth consistency, reduce ambiguity, and provide more complete 3D representations under communication constraints.
- CoCa3D integrates single-agent camera-only detection with collaborative depth estimation and collaborative detection feature learning.Co-Depth improves estimated depth, while Co-FL complements individual 3D features for more holistic detection.
- The framework formulates collaboration as maximizing detection performance across agents under a communication budget using informative, compact messages.Messages should include depth information for resolving depth ambiguity and detection clues for addressing occlusion and long-range limitations.
- Single-agent detection encodes images, estimates discretized depth distributions, projects features into 3D voxels, and collapses them into BEV features for decoding.The depth distribution weights voxel features before BEV construction, while the decoder predicts object classes, positions, sizes, and orientations.
- Co-Depth uses multi-view consistency to refine depth, sharing spatially sparse voxel features and depth probabilities selected from confident image locations.Low-uncertainty locations are selected because transmitting high-uncertainty features can introduce ambiguity and degrade single-agent depth estimation.
- Depth messages are fused using confidence-filtered matching across neighboring views, then a collaborative depth distribution reweights voxel features before BEV collapse.Low-confidence neighboring views are excluded, and the updated depth probability reflects both single-view confidence and cross-view consistency.
4. Experimental Results
Across real-world and simulation datasets, CoCa3D improves camera-only 3D detection through collaboration, with gains in accuracy, depth estimation, and performance-bandwidth trade-offs. Increasing the number of agents enables camera-only detection to match or exceed LiDAR in supported settings.
- Benchmark comparison: 44.21% on DAIR-V2X, 30.60% on OPV2V+, and 12.59% on CoPerception-UAVs+ improvement over previous SOTAs for AP@70.Table 1 reports the benchmark gains across the three datasets.
- Collaborative camera-only 3D detection overtakes LiDAR: With 10 collaborative agents, CoCa3D outperforms LiDAR at AP@0.5/0.7 on OPV2V+, while performance increases with the number of agents.The study also projects that camera-only detection could outperform LiDAR with 7 collaborative vehicles on DAIR-V2X.
- Multi-agent collaboration evaluation: Collaboratively estimated depth significantly outperforms single-agent depth and approaches ground-truth depth, improving camera-only detection toward its upper bound.Table 2 and the depth visualization support gains from collaborative depth estimation.
- Visualization of detection results: Collaborative depth and feature learning recover missed objects and improve holistic detection in long-range, occluded, and background regions.The qualitative results report gains from both Co-Depth and Co-FL, including cases where collaborative cameras outperform LiDAR.
- Trade-off between detection performance and communication cost: CoCa3D achieves a superior detection-performance and communication-cost trade-off over Where2comm across varying bandwidths.The method adapts communication by adjusting depth-uncertainty and detection-confidence thresholds.
- Effect of components in collaborative depth estimation: Co-Depth steadily improves depth accuracy, while linear-increasing depth spacing remains robust across supervision choices and stricter AP@80 evaluation.Uniform spacing fails AP@80 without dense supervision, whereas linear spacing uses more candidates in frequently occurring depth ranges.
5. Conclusion and Limitation
CoCa3D uses multi-agent collaboration to improve camera-only 3D detection and approach holistic 3D detection. The paper reports strong performance-bandwidth trade-offs and LiDAR-overcoming results in OPV2V+, while noting that validation relies mainly on simulation because real-world multi-agent data are scarce.
- CoCa3D introduces multi-agent collaboration to improve camera-based 3D detection toward holistic 3D perception.
- With sufficient collaborative agents, CoCa3D achieves a state-of-the-art perception-bandwidth trade-off and overtakes LiDAR-based detectors on OPV2V+.
- Real-world validation remains limited because DAIR-V2X is the sole public real-world multi-agent dataset, containing only one vehicle and one roadside unit.
6. Appendix
The appendix describes the simulation datasets, training configurations, object-distance analysis, and robustness evaluation used for CoCa3D. It also documents the simulated environments and multi-agent sensor setups for OPV2V+ and CoPerception-UAVs+.
- OPV2V+ extends OPV2V to approximately 10 collaborative agents, each equipped with four cameras and four depth sensors.
- CoPerception-UAVs+ extends its predecessor to approximately 10 agents by combining CARLA traffic scenes with AirSim drone simulation.
- The car detector uses LSS- and CaDDN-style implementations with 50 uniformly spaced depth categories, followed by separate single-agent and collaborative training stages.
- The appendix includes an object-distance distribution and ten-agent qualitative samples for both OPV2V+ and CoPerception-UAVs+.
- The drone detector follows DVDET with 10 uniformly spaced depth categories and a 140-epoch training schedule.
- CoCa3D remains robust under simulated pose errors and is reported as as robust as Where2comm in the corresponding figure.