UniMM-V2X: MoE-Enhanced Multi-Level Fusion for End-to-End Cooperative Autonomous Driving

Introduction

• Standalone end-to-end driving remains constrained by sensor range and difficulty handling rare critical events and other agents’ intentions, motivating V2X cooperation.The introduction identifies real-time information exchange as a key role for V2X communication.Source: Introduction · p. 1
• Multi-level cooperation exchanges information at both perception and prediction levels so agents can reason jointly about scene understanding and future behaviors.The framework transmits track, map, and occupancy queries or maps for perception, then motion queries for prediction.Source: Figure 1 · p. 1, Introduction · p. 2
• MoE-enhanced encoders dynamically generate task-specialized BEV representations for heterogeneous perception, prediction, and planning requirements.This addresses the limitation that a shared BEV encoder may not satisfy all downstream tasks equally.Source: Introduction · p. 2
• MoE-equipped decoders use expert branches for distinct motion patterns such as going forward, turning left, and turning right.The motion-specific queries complement multi-level fusion and are intended to improve decision quality, interpretability, and reliability.Source: Introduction · p. 2, Figure 3 · p. 3
• UniMM-V2X achieves SOTA results in perception, prediction, and planning compared with single-agent and existing cooperative methods.The paper attributes this outcome to combining multi-level fusion with MoE-based specialization.Source: Introduction · p. 2

End-to-End Autonomous Driving

• End-to-end driving methods unify intermediate tasks or reduce computational cost, but the cited approaches remain limited to single-agent inputs.UniMM-V2X extends this paradigm with cross-agent communication for cooperative planning.Source: End-to-End Autonomous Driving · p. 3

Cooperative Autonomous Driving

• Existing cooperative-driving methods use learned communication or unified frameworks, yet many rely on vanilla fusion or stop cooperation at perception.UniMM-V2X instead supports cooperative reasoning from spatial observations to motion intents.Source: Cooperative Autonomous Driving · p. 3

Mixture of Experts

• MoE methods use conditional computation and sparse expert activation, but prior autonomous-driving applications typically restrict MoE to a single stage.The cited prior work includes Transformer scaling and sensor-scheduling or action-guidance applications.Source: Mixture of Experts · p. 3
• UniMM-V2X integrates MoE into both the BEV encoder and motion decoder to produce task-specialized representations and expert-guided motion predictions.This design targets heterogeneous task demands in multi-agent cooperative driving.Source: Mixture of Experts · p. 3

Overview

• UniMM-V2X combines perception-level and prediction-level cooperation with MoE-enhanced encoding and decoding for unified end-to-end decision-making.The framework includes cooperative perception, motion prediction, and planning modules, with fusion across agents at perception and prediction levels.Source: Overview · p. 3, MoE for Adaptive Feature and Motion… · p. 3

MoE for Adaptive Feature and Motion Modeling

• MoE replaces conventional FFNs in the BEV encoder and motion decoder, using selected experts to generate task-adaptive features and motion queries.The decoder’s expert branches model distinct motion patterns, while load balancing encourages more uniform expert activation.Source: MoE for Adaptive Feature and Motion… · p. 4

Multi-Agent Perception-Level Fusion

• Perception-level fusion dynamically combines track, map, and occupancy information, using spatial embeddings and attention to model inter-agent relationships.TrackFusion transforms other agents’ queries into the ego frame and incorporates reference points as spatial contextual priors; map and occupancy fusion use MLP and max operations, respectively.Source: Multi-Agent Perception-Level Fusion · p. 4

Cross-View Prediction-Level Fusion

• Prediction-level fusion aligns transmitted motion queries spatially, concatenates them with ego queries, and applies attention to support cooperative motion reasoning.TrajFusion uses agent-level anchors, rotation-based coordinate transformation, position embedding, and attention over fused perception context.Source: Cross-View Prediction-Level Fusion · pp. 4–5

Learning

• The training objective jointly optimizes tracking, mapping, occupancy, motion prediction, planning, and MoE load balancing in an end-to-end manner.The loss contains six components: Ltrack, Lmap, Locc, Lmot, Lplan, and Lmoe.Source: Learning · p. 5

Main Results

• 33.2% lower planning L2 error and 52.0% lower collision rate than UniV2X establish the strongest overall planning result.UniMM-V2X achieves 1.49m average L2 error and 0.12% average collision rate, outperforming all evaluated baselines despite slightly higher communication overhead.Source: Main Results · p. 6
• 39.7% higher mAP and 77.2% higher AMOTA than UniV2X demonstrate substantial cooperative perception gains without additional perception-level communication cost.Occupancy IoU-n also improves by +0.8% over UniV2X.Source: Main Results · p. 6
• 7.2% lower minADE and 6.8% lower minFDE than UniV2X yield the best reported motion-prediction performance, with 0.64m minADE, 0.69m minFDE, and 13.2% MissRate.These prediction improvements contribute to the final planning gains.Source: Main Results · p. 6, Table 4 · p. 6

Ablation Study

• Single-level fusion improves its aligned tasks but leaves other driving stages limited, whereas multi-level fusion propagates high-quality intermediate features across all modules.Perception-level fusion mainly improves detection and tracking, while prediction-level fusion improves planning safety for occluded or uncertain trajectories.Source: Ablation Study · p. 6, Table 5 · p. 6
• Applying MoE to both the BEV encoder and motion decoder performs best because task-aware features and expert motion specialization jointly support complex driving behaviors.Encoder-only MoE improves perception and planning, while decoder-only MoE provides limited gains.Source: Ablation Study · p. 6, Table 6 · p. 7
• Eight motion-decoder experts provide the best trade-off, while too few reduce specialization and too many cause data sparsity and under-utilization.MoE benefits are weaker in the perception decoder because limited token diversity can introduce gating noise.Source: Ablation Study · p. 7, Table 6 · p. 7

Practicality and Reliability of the System

• 87.9× lower communication cost than bandwidth-heavy BEV methods and 5.4 FPS support practical deployment, while UniMM-V2X remains superior across variable bandwidth settings.Inference is slightly slower than UniV2X’s 5.8 FPS, but the reported planning and safety improvements are presented as justifying the added cost.Source: Practicality and Reliability of the… · p. 7, Figure 5 · p. 7

Conclusion

• UniMM-V2X reports +39.7% detection, +77.2% tracking, -7.2% motion prediction error, -33.2% L2 error, and -52.0% collision rate versus the previous SOTA method.The framework also maintains comparable communication cost and demonstrates reliability under different bandwidth constraints.Source: Conclusion · p. 7

MoE Settings

• The encoder and decoder each use six MoE layers with eight experts, top-2 routing, and load-balancing loss weight λ = 0.03.This configuration is intended to support adaptive specialization while controlling expert utilization.Source: Implementation Details · p. 10

Perception

• Perception uses a six-layer decoder, a 50-meter radius, and a 0.25 tracking threshold to retain robustness under occlusion or partial views.The threshold is the minimum matching confidence for associating objects across consecutive frames.Source: Implementation Details · p. 10
• Perception-level fusion combines track, map, and occupancy information through TrackFusion, MapFusion, and OccFusion components.Map queries are fused with an MLP, while aligned occupancy maps use a max operation with τ = 0.1.Source: Implementation Details · p. 10

Prediction

• Prediction defines six motion modes per target and uses four past steps to predict the next twelve future steps.The multiple modes represent uncertainty and multimodal future behavior.Source: Implementation Details · p. 10

Comparative Experiments

• Comparisons span DAIR-V2X and V2X-Sim, with no-, early-, late-, and intermediate-fusion strategies and representative cooperative and single-agent baselines.Planning baselines include UniAD, VAD, SparseDrive, V2VNet, CooperNaut, and UniV2X, with UniV2X results reproduced from official checkpoints.Source: Implementation Details · p. 10

Training

• Training proceeds through infrastructure perception pretraining, ego perception pretraining, cooperative perception training, and frozen-perception prediction and planning training followed by fine-tuning.The staged process separates perception learning from later cooperative prediction and planning optimization.Source: Implementation Details · p. 10

• Six task families are evaluated alongside transmission cost: detection, tracking, online mapping, occupancy prediction, motion prediction, and planning.Source: Evaluation Metrics · pp. 10–11

Object Detection and Multi-Object Tracking

• Detection is measured by mAP, while tracking uses AMOTA to capture detection quality and association accuracy over recall levels.Detection and tracking focus on the safety-critical Car class.Source: Evaluation Metrics · p. 11

Online Mapping and Occupancy Prediction

• Online mapping and occupancy prediction use IoU, with occupancy evaluated separately over near 30 × 30m and far 50 × 50m spatial ranges.These metrics assess semantic map overlap and occupied-versus-free-space segmentation.Source: Evaluation Metrics · p. 11

Motion Prediction

• Motion prediction uses minADE, minFDE, and miss rate to measure trajectory deviation and failures across predicted modes.Miss rate counts ground-truth trajectories deviating beyond threshold δ, typically 2m, from all predicted modes.Source: Evaluation Metrics · p. 11

Planning

• Planning uses L2 error and collision rate at 1s, 2s, and 3s horizons to assess trajectory accuracy and safety.The protocol follows ST-P3.Source: Evaluation Metrics · p. 11

Transmission Cost

• Communication efficiency is quantified by bytes per second under 2 Hz transmission using serialized 32-bit features and queries.Calibration matrices and ego-pose data are excluded; motion-prediction fusion causes a slight overhead increase over UniV2X.Source: Evaluation Metrics · p. 11

Dataset

• V2X-Sim provides 100 simulated urban scenes with connected vehicles, cameras, LiDAR, HD maps, and annotations for cooperative driving tasks.The dataset supports detection, tracking, motion prediction, and planning evaluation.Source: Results on V2X-Sim Dataset · p. 11

Experiment Settings

• The experiments use two vehicles with front-view cameras, splitting 80 scenes for training, 10 for validation, and 10 for testing.Planning is evaluated at standard 1s, 2s, and 3s intervals for consistency with field practice.Source: Results on V2X-Sim Dataset · pp. 11–12

• 20.39% lower L2 error and 7.95% lower collision rate than UniV2X improve planned-trajectory accuracy and driving safety.Planning results are summarized in Table 7.Source: Results on V2X-Sim Dataset · p. 12, Table 7 · p. 12
• 15.8% lower motion prediction error than UniV2X improves the trajectory estimates used for planning.Motion prediction results are summarized in Table 8.Source: Results on V2X-Sim Dataset · p. 12, Table 8 · p. 12

• The comparison reports GPU memory, frames per second, and total transmitted data to characterize real-time performance and communication trade-offs.Source: Efficiency of the System · p. 12
• UniMM-V2X increases GPU memory usage by approximately 2.9% and reduces FPS by 0.47 versus UniV2X.The efficiency comparison is summarized in Table 9.Source: Efficiency of the System · p. 12, Table 9 · p. 12
• The limited efficiency gain from sparse MoE is attributed to the model not yet being large enough to distribute computation across many experts as effectively as large VLMs.Source: Efficiency of the System · p. 12
• MoE improves representation learning for perception and prediction, while enriched features and motion queries slightly increase communication overhead.The paper frames these costs as supporting gains in prediction robustness and planning safety.Source: Efficiency of the System · p. 12

• Figure 6 presents UniMM-V2X planning performance on DAIR-V2X across left-turn, straight-driving, and right-turn behaviors.The visualization is intended to illustrate effects across perception, prediction, and final planning.Source: Qualitative Visualization · p. 12, Figure 6 · p. 13