Source-linked AI summary
CoCam4D: Geometry-Aware Cooperative 4D Perception for Camera-Only Autonomous Driving
Soham Pahari, Sudip Das, Arindam Das, Ujjwal Bhattacharya
TL;DR
Camera-only collaborative perception must contend with distance-dependent monocular depth uncertainty. CoCam4D uses feedforward Gaussian scene representations with explicit uncertainty for Bayesian multi-agent fusion. The paper reports that five cooperating camera agents can achieve performance comparable to single-vehicle LiDAR, with reported gains on vision-only and real-world benchmarks.
Problem
Camera-only perception remains limited by monocular depth uncertainty, which increases quadratically with object depth.
Source: §1 · p. 2Method
CoCam4D reconstructs velocity-aware 3D Gaussian scenes with uncertainty per vehicle and fuses them across agents using an information filter.
Source: §1 · p. 3Results
Five cooperating camera agents achieve performance comparable to a single-vehicle LiDAR system, and CoCam4D improves AP@0.5 by 10.6% over CoCa3D on DAIR-V2X-C.
Source: §1 · p. 3, §5.2 · p. 12Takeaways & Limitations
The reported cooperative regime suggests camera-only systems can match or exceed single-vehicle LiDAR performance.
Source: §6 · p. 14The formulation considers connected vehicles within V2V range, each observing consecutive frames from multiple cameras.
Source: §3 · p. 4
Abstract
from arXiv · showhide
Autonomous vehicles often suffer from limited perception due to occlusions, blind spots, limited sensor range, and the complex nature of surrounding environments. Multi-agent collaborative perception (CP) addresses these challenges by allowing vehicles to share sensory information and reconstruct the scene cooperatively. However, camera-only perception remains fundamentally limited by the uncertainty of distance-dependent monocular depth estimation. We propose CoCam4D, a Bayesian framework for collaborative perception that explicitly models geometric uncertainty. It uses a VGGT-based feedforward network to generate 3D Gaussian scene representations with associated uncertainty estimates, enabling multiple vehicles or agents to efficiently combine their observations. By sharing compact Gaussian primitives, reliable observations from one agent can reduce the depth uncertainty of another without requiring LiDAR sensors. To support real-world deployment, we introduce Dynamic Object Primitives (DOPs), a compact 35-byte representation designed for efficient C-V2X communication. Extensive experiments show that our proposed method consistently outperforms recent vision-only methods, achieving improvements of 11.48% on OPV2V+ and 10.62% on DAIR-V2X-C, demonstrating the potential of geometrically grounded collaborative perception for LiDAR-free autonomous driving.
1 Introduction
CoCam4D frames camera-only collaboration as a way to address monocular geometry limits, using explicit uncertainty and Bayesian fusion to combine velocity-aware Gaussian scene representations. Its introduction also highlights compact communication and reports a five-agent comparison with single-vehicle LiDAR.
1 Introduction
- V2X lets connected agents exchange perceptual information to cooperatively understand driving scenes beyond a single vehicle’s capability.The introduction connects shared perception with more complete and reliable traffic-scene understanding.Source: §1 · p. 1
- Camera-based cooperative methods often share compressed BEV features without uncertainty modeling, while monocular depth uncertainty grows quadratically with object depth.The introduction identifies principled Bayesian fusion of distance-dependent, anisotropic uncertainty as underexplored.Source: §1 · p. 2
- CoCam4D reconstructs velocity-aware 3D Gaussian scenes per vehicle and combines them across agents with principled information-filter fusion.The framework propagates uncertainty from per-pixel depth error to Gaussian covariance and fusion weighting.Source: §1 · p. 3
- Dynamic Object Primitives encode object state and observation covariance in approximately 35 bytes per object for V2V communication.The introduction describes this compact representation as fitting one C-V2X packet.Source: §1 · p. 3
- Five cooperating camera agents achieve performance comparable to a single-vehicle LiDAR system.The contribution summary presents this as a cooperative-density result.Source: §1 · p. 3
2 Related Work
Related work spans camera-only 3D understanding, cooperative perception, feedforward 4D reconstruction, Gaussian scene representations, and cooperative tracking. The section contrasts these lines with CoCam4D’s explicit uncertainty-aware cooperative Gaussian propagation.
Camera-Only 3D Perception
Cooperative Perception
- Cooperative perception ranges from raw-data early fusion to object-level late fusion and bandwidth-efficient intermediate feature fusion, while vision-based methods often omit uncertainty modeling.Intermediate fusion is described as dominant; camera-only approaches face monocular depth ambiguity and multi-view misalignment.Source: §2 · pp. 3–4
Feedforward 4D Reconstruction
3D Gaussians for Driving and Collaboration
Cooperative Multi-Object Tracking
- Cooperative tracking methods use discrete bounding boxes and simplified uncertainty assumptions, limiting their fit to camera-only perception with anisotropic depth uncertainty.The section cites DMSTrack and MOT-CUP [33] as examples.Source: §2 · p. 4
3 Preliminaries
The preliminaries formulate cooperative 4D perception for connected vehicles observing consecutive multi-camera frames. CoCam4D breaks the task into per-vehicle reconstruction, compact messaging, and uncertainty-weighted fusion.
3 Preliminaries
- Each vehicle’s output world model contains dynamic-agent position, velocity, heading, and dimensions.The setup considers N connected vehicles within V2V range, each observing F consecutive frames from C cameras.Source: §3 · p. 4
- Per-vehicle reconstruction estimates 3D Gaussians, velocities, and depth uncertainty in a single forward pass.The encoder maps each vehicle’s observations to its reconstructed scene quantities.Source: §3 · p. 5
- Compact cooperative messaging packages each object’s state and observation covariance as a Dynamic Object Primitive.The covariance is derived analytically from predicted uncertainty and camera geometry, with approximately 35 bytes per object.Source: §3 · p. 5
- The information filter fuses received primitives by confidence, using anisotropic covariance so lateral views can compensate for head-on depth uncertainty.The formulation contrasts this with sharing dense features or boxes that discard uncertainty.Source: §3 · pp. 4–5
4 Proposed Method
CoCam4D reconstructs velocity-aware Gaussian scenes independently at each vehicle, compresses object representations into DOP messages, then fuses them geometrically using observation uncertainty. The method connects per-pixel depth uncertainty to Gaussian covariance and cooperative weighting.
4 Proposed Method
- The proposed pipeline reconstructs velocity-aware Gaussian scenes per vehicle, extracts bandwidth-limited DOP messages, and fuses observations according to complementary viewing geometry.The three stages are per-vehicle reconstruction, DOP extraction, and information-filter fusion.Source: §4 · p. 5
4.1 Per-Vehicle Feedforward 4D Reconstruction
- Each vehicle reconstructs a dynamic scene in one forward pass, predicting metric depth, uncertainty, velocity, and Gaussian primitives from multi-frame camera features.The depth uncertainty expands Gaussian extent along the viewing ray and propagates to covariance used in fusion weighting; causal temporal attention supports directed motion.Source: §4.1 · pp. 5–7
4.2 Dynamic Object Primitive Extraction
- DOPs encode an object's centroid, velocity, heading, dimensions, and observation covariance in about 35 bytes while retaining geometric confidence.Their anisotropic uncertainty follows camera geometry: depth uncertainty grows quadratically with distance, while lateral uncertainty grows linearly.Source: §4.2 · p. 8
4.3 Cooperative Fusion
- CoCam4D combines vehicle observations through geometric alignment and information-filter fusion, with covariance-weighted updates and Gaussian union for more complete object shapes.Received DOPs are motion-compensated for communication delay; fusion requires no learned calibration network or multi-agent training.Source: §4.3 · pp. 8–10
5 Experiments
Experiments span simulated OPV2V+ and real-world DAIR-V2X-C, measuring detection, reconstruction, tracking, communication cost, and end-to-end latency.
5.2 Experimental Results
- 11.5% AP@0.5 at 0–50 m and 11.3% at 50–100 m over CoCa3D; at N=5, CoCam4D exceeds single-vehicle LiDAR across reported metrics.The N=2 comparison also uses 21× less bandwidth than CoCa3D.Source: Table 2 · p. 11, §5.2 · p. 12
- 10.6% AP@0.5 over CoCa3D on DAIR-V2X-C, with generalization to vehicle-infrastructure cooperation despite training exclusively on V2V data.The passage conditions this generalization on well-calibrated observation covariance.Source: Table 3 · p. 12, §5.2 · p. 12
- At N=5 on CARLA, CoCam4D reaches 63.7% AP@0.5, 40.2% AP@0.7, and 47.8 AMOTA, exceeding single-vehicle LiDAR on all three metrics.Performance scales nearly linearly through N=5, with diminishing returns beyond N=10 as views become more redundant.Source: Table 4 · p. 12, §5.2 · p. 12
5.3 Ablation Studies
- Removing the uncertainty head causes the largest ablation degradation (5.2 AP, 4.5 AMOTA), while symmetric global attention reduces AMOTA by 7.2 and AP by 3.6.The authors link the uncertainty-head result to depth-dependent fusion weighting and the attention result to temporal velocity consistency.Source: §5.3 · p. 13
- Removing lifespan lowers AP by 2.3; removing velocity from association lowers AP by 1.8 and AMOTA by 5.4, particularly affecting identity consistency at intersections.The lifespan ablation is attributed to stale static Gaussian accumulation; DOP-driven dynamic detection remains largely unaffected.Source: §5.3 · p. 13
- Replacing Gaussian product fusion with learned BEV attention reduces AP@0.5 by 6.7 and raises bandwidth from 58 to 186 KB/s; late box fusion reduces AP by 4.5.The ablation is conducted on OPV2V+ with N=3 cooperative vehicles.Source: §5.3 · p. 13, Table 5 · p. 13
- Under 500 ms latency, CoCam4D degrades by 2.7 AP versus 11.4 for CoCa3D and 11.5 for Where2comm; at 1.0 m pose-noise STD, it retains 55.4 AP versus CoCa3D’s 43.2.The authors attribute robustness to velocity-based forward prediction and covariance inflation in fusion.Source: Table 6 · p. 13, §5.3 · p. 14
5.4 Communication Efficiency
- At N=2, DOP-only reaches 52.8 AP at 7 KB/s; the full system reaches 57.3 AP at 58 KB/s and a 24× higher AP/KB ratio than CoCa3D.The full system combines cached static Gaussians with per-frame DOPs.Source: Table 7 · p. 14, §5.4 · p. 14
6 Conclusion
CoCam4D addresses monocular geometry limits through uncertainty-aware Bayesian fusion of 3D Gaussian scene representations. Its compact Dynamic Object Primitives support communication under bandwidth constraints, and the paper reports a cooperative regime where camera-only systems match or exceed single-vehicle LiDAR.
6 Conclusion
- CoCam4D represents scenes as 3D Gaussian primitives with explicit covariances for Bayesian multi-agent fusion, without LiDAR supervision or iterative optimization.The framework integrates observations in a single forward pass.Source: §6 · p. 14
- Dynamic Object Primitives provide a compact 4D representation for uncertainty-aware communication under bandwidth constraints.Source: §6 · p. 14
- The paper reports a cooperative regime where camera-only systems match or exceed single-vehicle LiDAR performance.Source: §6 · p. 14
- CoCam4D points toward uncertainty-aware collaboration for LiDAR-free autonomous driving and joint perception and planning systems.Source: §6 · p. 14