Source-linked AI summary
DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Driving
Zhenjie Yang, Yilin Chai, Xiaosong Jia, Qifeng Li, Yuqian Shao, Xuekai Zhu, Haisheng Su, Junchi Yan
TL;DR
End-to-end autonomous driving still struggles with redundant multi-view processing and diverse, rare driving behaviors. DriveMoE extends Drive-π0 with scene-specialized vision and skill-specialized action MoE modules, achieving state-of-the-art performance on Bench2Drive while supervised view selection remains a scope boundary.
Problem
Existing VLA driving models process multi-view inputs inefficiently and use unified policies that inadequately address rare, context-dependent maneuvers.
Method
DriveMoE extends Drive-π0 with a router-based Vision MoE that selects relevant cameras and an Action MoE that activates skill-specific planning experts.
Results
DriveMoE achieves state-of-the-art performance on Bench2Drive and substantially improves driving performance over Drive-π0.
Takeaways & Limitations
Combining dynamic visual selection with explicit behavioral specialization improves efficiency, robustness, and handling of rare safety-critical driving scenarios.
Takeaways & Limitations
Vision MoE currently relies on supervised camera-view annotations, while unsupervised view selection is left as a promising direction.
Abstract
from arXiv · showhide
End-to-end autonomous driving (E2E-AD) demands effective processing of multi-view sensory data and robust handling of diverse and complex driving scenarios, particularly rare maneuvers such as aggressive turns. Recent success of Mixture-of-Experts (MoE) architecture in Large Language Models (LLMs) demonstrates that specialization of parameters enables strong scalability. In this work, we propose DriveMoE, a novel MoE-based E2E-AD framework, with a Scene-Specialized Vision MoE and a Skill-Specialized Action MoE. DriveMoE is built upon our $π_0$ Vision-Language-Action (VLA) baseline (originally from the embodied AI field), called Drive-$π_0$. Specifically, we add Vision MoE to Drive-$π_0$ by training a router to select relevant cameras according to the driving context dynamically. This design mirrors human driving cognition, where drivers selectively attend to crucial visual cues rather than exhaustively processing all visual information. In addition, we add Action MoE by training another router to activate specialized expert modules for different driving behaviors. Through explicit behavioral specialization, DriveMoE is able to handle diverse scenarios without suffering from modes averaging like existing models. In Bench2Drive closed-loop evaluation experiments, DriveMoE achieves state-of-the-art (SOTA) performance, demonstrating the effectiveness of combining vision and action MoE in autonomous driving tasks. We will release our code and models of DriveMoE and Drive-$π_0$.
1. Introduction
Existing VLA driving models face redundant multi-view processing and insufficient specialization for rare behaviors. DriveMoE addresses these limitations with dynamic camera selection and skill-specific action experts built on Drive-π0.
- Closed-loop end-to-end driving remains unsatisfactory despite progress on open-loop benchmarks.
- Vanilla vision processors process every camera view, creating redundant representations and substantial computational overhead.
- Query-based vision processors reduce visual tokens but can lose geometric and positional information while requiring additional pre-training.
- Unified policy networks favor frequent scenarios and inadequately address rare maneuvers such as emergency braking and aggressive turning.
- MoE specialization remains under-explored in autonomous-driving vision and action domains, motivating DriveMoE’s dual-module design.
- DriveMoE combines a Scene-Specialized Vision MoE for context-relevant camera selection with a Skill-Specialized Action MoE for specialized planning.
2. Method
DriveMoE extends Drive-π0 with scene-specialized vision routing and skill-specialized action experts for end-to-end autonomous driving. The framework dynamically selects camera views and driving-behavior experts to reduce visual redundancy and preserve diverse trajectory modes.
- Scene-Specialized Vision MoE: Dynamic camera selection reduces visual-token load while retaining contextually relevant multi-view information for driving decisions.Camera-view annotations are derived from future trajectories, bounding boxes, and maps, with router training using annotated binary selection labels.
- Scene-Specialized Vision MoE: The Vision MoE router uses front-view embeddings and future goal waypoints to assign camera-view selection probabilities before expensive backbone computation.Unselected views can be skipped, and selected views are supplied to the VLM alongside fixed views and text.
- Skill-Specialized Action MoE: The Action MoE replaces decoder FFNs with skill-specific experts and conditionally routes inputs to preserve distinct behavior modes, including rare maneuvers.The design targets behaviors such as acceleration, braking, turning, merging, overtaking, emergency braking, yielding, and traffic-sign responses.
- Skill-Specialized Action MoE: Token-level Action MoE routes each trajectory token independently, whereas trajectory-level Action MoE selects experts after averaging the entire token sequence.Token-level routing models short-horizon subtasks; trajectory-level routing treats each trajectory as one scenario or driving skill and uses skill labels for router supervision.
3. Experiments
DriveMoE is evaluated on Bench2Drive using closed-loop driving metrics, capability dimensions, and ablations of its Vision MoE and Action MoE components. The experiments show SOTA performance, benefits from dynamic supervised view selection, and complementary contributions from both MoE modules.
- 3.1. Datasets & Benchmark & Metric: DriveMoE is evaluated in CARLA on the Bench2Drive benchmark using 220 official routes and official metrics.All results are averaged over three runs.
- 3.2. Comparison with State-of-the-Art Methods: 22.8% driving-score improvement and 62.1% success-rate improvement over Drive-π0 establish SOTA closed-loop performance on Bench2Drive.The method also achieves the lowest reported L2 error on the open-loop metric, which the authors describe primarily as a convergence indicator.
- 3.2. Comparison with State-of-the-Art Methods: SOTA results are obtained across five key driving capabilities and their overall average.The evaluated dimensions include Merging, Overtaking, Emergency Braking, Yielding, and Traffic Signs.
- 3.1. Datasets & Benchmark & Metric: Driving Score combines Route Completion with Infraction Score, while Success Rate measures routes completed within time without traffic violations.Bench2Drive also reports Efficiency and Comfort alongside capability-specific evaluations.
- 3.3. Ablation Study: Adding individual fixed camera views provides moderate improvements, whereas adding multiple views increases visual tokens, training difficulty, and latency.The multi-view configurations in Exp 5–7 degrade performance as latency increases.
- 3.3. Ablation Study: Dynamic view selection improves driving score and success rate, with explicit supervision further enhancing both metrics.This comparison covers unsupervised dynamic selection in Exp 8 and supervised selection in Exp 9, corresponding to DriveMoE.
- 3.3. Ablation Study: Trajectory-level Action MoE consistently outperforms token-level routing and is therefore used as the default Action MoE.The Action MoE ablations also compare configurations with different numbers of non-shared experts.
- 3.3. Ablation Study: Removing either Vision MoE or Action MoE noticeably reduces driving score and success rate, indicating complementary contributions from both modules.DriveMoE substantially improves driving performance relative to Drive-π0.
4. Conclusion
DriveMoE extends Drive-π0 with MoE specialization in both vision and action for end-to-end autonomous driving. It dynamically selects relevant camera views and behavior-specific experts, achieving SOTA Bench2Drive performance while improving computational efficiency and robustness to rare, safety-critical scenarios.
- 4. Conclusion: DriveMoE integrates Scene-Specialized Vision MoE and Skill-Specialized Action MoE into the Drive-π0 end-to-end driving framework.The vision component dynamically selects relevant camera views, while the action component activates experts tailored to driving behaviors.
- 4. Conclusion: DriveMoE achieves state-of-the-art performance on Bench2Drive while enhancing computational efficiency and robustness to rare, safety-critical driving scenarios.
5. Related Work
Prior VLM and VLA approaches improve reasoning and continuous-action modeling for autonomous driving, but MoE-based specialization remains unexplored for diverse driving domains.
- VLM-based driving methods support generalization, open-set reasoning, scalability, explanation, graph-based reasoning, interpretability, or knowledge transfer.
- Existing approaches predominantly use discrete tokens for driving policies, leaving continuous tokenization and diffusion-based policies underexplored.
- Continuous-action VLA methods such as OpenVLA, Diffusion Policy, and π0 model continuous action distributions through sequence prediction and global optimization.
- Sparse MoE architectures improve LLM task specialization and representation capacity while maintaining inference efficiency through conditional computation.
- The paper presents the first end-to-end autonomous-driving framework integrating MoE at both vision and action levels.
6. Conditional Flow Matching Loss
The method predicts future action trajectories through conditional flow matching, training a denoising flow over noisy actions to model continuous trajectory distributions.
- The model predicts future action trajectories in a denoising manner using a conditional flow matching loss.
- Flow-matching subscripts denote action timesteps, superscripts denote flow-matching timesteps, and τ ranges from 0 to 1.
- Noisy actions are sampled as A^τ_t = τA_t + (1 − τ)ϵ, and the network is trained to output the denoising flow vθ(A^τ_t|A_t) = ϵ − A_t.
- This formulation learns the underlying trajectory distribution through a continuous stochastic process rather than pointwise supervision.
- The approach is suited to multimodal or uncertain planning scenarios where smooth action trajectories are important.
7. Implementation Details
Implementation uses camera-view annotations and fixed skill definitions to train Drive-π0 and DriveMoE, while PID control converts predicted waypoints into vehicle commands.
- Routing annotations: Vision-routing annotations add camera-view importance labels to Bench2Drive using an inexpensive and straightforward procedure.
- Routing annotations: The action-routing setup defines five driving skills: Merging, Overtaking, Emergency Brake, Give Way, and Traffic Sign.
- Model inputs: Drive-π0 uses two sequential front-view images and current plus historical vehicle state to predict 10 future waypoints.
- Model inputs: DriveMoE combines two sequential front-view images with the Top-1 camera view selected dynamically by the vision router.
- Vehicle control: The desired speed uses the 7th predicted waypoint, while steering uses the 10th waypoint for stable and responsive control.
8. Discussion on Unsupervised View Selection and Knowledge Distillation
The discussion identifies unsupervised view selection and knowledge distillation as future directions for reducing annotation costs, improving generalization, and supporting deployment.
- Unsupervised view selection: Unsupervised camera-view learning could further reduce annotation costs and improve generalization.
- Unsupervised view selection: The current supervised annotations use inexpensive heuristics from trajectories and maps to provide stable initial training for Vision MoE.
- Unsupervised view selection: Image-level token pruning after view selection is identified as a potential extension informed by DySS and LightVLA.
- Knowledge distillation: Knowledge distillation is proposed as a future direction in which DriveMoE serves as a teacher for skill-specific compact students.
9. Annotation for Router
DriveMoE uses Bench2Drive annotations and scenario-specific rules to supervise its vision and action routers. The rules select camera views based on driving context, while the action router follows Bench2Drive’s skill definitions.
- Vision Router: Bench2Drive annotations supervise context-dependent camera selection for DriveMoE’s Vision Router.The rules identify special driving scenarios and annotate relevant camera views.
- Vision Router: Lane direction determines whether the router annotates a rear-side or front-side camera when the ego-vehicle changes or occupies a target lane.The corresponding rear-side camera is selected for same-direction movement, while the front-side camera is selected for temporary occupation of the opposing lane.
- Vision Router: Highway merging and vehicle cut-in scenarios use the ego-vehicle’s lane position and junction distance to select the side camera facing the merge.
- Vision Router: Emergency-vehicle scenarios annotate the camera facing the approaching high-speed vehicle.
- Action Router: The Action Router adopts Bench2Drive’s division of 44 scenarios into 5 skills.
10. Experiment on nuScenes
The nuScenes experiment reports competitive planning accuracy and lower collision rates for DriveMoE, while noting that open-loop evaluation does not fully reflect driving performance. The model’s added inference cost is attributed primarily to an additional camera view and Top-3 expert activation.
- Performance: DriveMoE achieves competitive L2 error while significantly reducing collision rate in the nuScenes open-loop planning evaluation.
- Performance: The results are described as demonstrating strong generalization to real-world scenarios.
- Evaluation caveat: Open-loop evaluation cannot fully reflect driving performance.
- Efficiency: The increased inference cost primarily comes from processing one additional camera view and activating Top-3 experts.