Source-linked AI summary
ARTEMIS: Autoregressive End-to-End Trajectory Planning with Mixture of Experts for Autonomous Driving
Renju Feng, Ning Xi, Duanfeng Chu, Rukang Wang, Zejian Deng, Anzheng Wang, Liping Lu, Jinxiang Wang, Yanjun Huang
TL;DR
ARTEMIS addresses the limited temporal modeling of static one-shot end-to-end driving and the difficulty of representing diverse driving behavior with a single network. It sequentially generates trajectory waypoints while routing scene-specific queries through specialized MoE experts, achieving competitive NAVSIM performance including 87.0 PDMS and 83.1 EPDMS with ResNet-34.
Problem
Static one-shot end-to-end inference inadequately captures environmental evolution, while singular architectures and ambiguous guidance limit modeling of diverse driving behaviors.
Method
ARTEMIS combines autoregressive waypoint generation, dynamically routed Mixture-of-Experts planning, perception, trajectory refinement, and expert-based batch reallocation.
Results
87.0 PDMS and 83.1 EPDMS are achieved on NAVSIM with a ResNet-34 backbone, with state-of-the-art results reported on extended metrics.
Takeaways & Limitations
Sequential planning and dynamically specialized experts provide a framework for modeling evolving trajectories and diverse driving environments within end-to-end autonomous driving.
Abstract
from arXiv · showhide
This paper presents ARTEMIS, an end-to-end autonomous driving framework that combines autoregressive trajectory planning with Mixture-of-Experts (MoE). Traditional modular methods suffer from error propagation, while existing end-to-end models typically employ static one-shot inference paradigms that inadequately capture the dynamic changes of the environment. ARTEMIS takes a different method by generating trajectory waypoints sequentially, preserves critical temporal dependencies while dynamically routing scene-specific queries to specialized expert networks. It effectively relieves trajectory quality degradation issues encountered when guidance information is ambiguous, and overcomes the inherent representational limitations of singular network architectures when processing diverse driving scenarios. Additionally, we use a lightweight batch reallocation strategy that significantly improves the training speed of the Mixture-of-Experts model. Through experiments on the NAVSIM dataset, ARTEMIS exhibits superior competitive performance, achieving 87.0 PDMS and 83.1 EPDMS with ResNet-34 backbone, demonstrates state-of-the-art performance on multiple metrics.
I. INTRODUCTION
ARTEMIS addresses limitations of modular and static one-shot end-to-end driving by combining sequential trajectory generation with dynamically routed specialized experts. It achieves 87.0 PDMS and 83.1 EPDMS on NAVSIM with a ResNet-34 backbone.
- Motivation: Traditional modular systems separate perception, prediction, and planning, while end-to-end models avoid these interfaces but often use static one-shot inference.Modular pipelines face cumulative errors and interdependencies; static end-to-end inference inadequately captures environmental evolution.
- Motivation: Autoregressive planning generates trajectory waypoints sequentially, preserving temporal dependencies and supporting decisions based on previously planned segments.ARTEMIS applies this strategy to model evolving trajectory context.
- ARTEMIS: ARTEMIS integrates perception, autoregressive MoE planning, trajectory refinement, and batch reallocation for expert activation patterns.The planning module dynamically selects experts for the current driving scenario before refining the generated trajectory.
- ARTEMIS: MoE dynamically routes scene-specific inputs to specialized experts, addressing ambiguous guidance and the representational limits of a single network across diverse driving scenarios.The approach models diverse behavioral patterns without relying on predefined guidance signals.
- Results: 87.0 PDMS and 83.1 EPDMS are achieved on NAVSIM with an identical ResNet34 backbone under standard and extended metrics.The reported results are from the large-scale real-world NAVSIM dataset.
II. METHOD
The method formulates end-to-end driving as conditional sequence generation from historical sensor observations to future trajectory waypoints. Autoregressive factorization and MoE routing provide the mathematical basis for sequential prediction with shared and specialized experts.
- End-to-End Autonomous Driving: End-to-end autonomous driving learns p(Y|S), mapping historical sensor observations S to a future trajectory sequence Y.S contains T historical observations, while Y contains H future waypoints with dimensionality d.
- End-to-End Autonomous Driving: Autoregressive modeling decomposes trajectory generation into conditional predictions of each waypoint given earlier generated points.The prior trajectory points are represented by y<t.
- End-to-End Autonomous Driving: A feature extractor maps sensor data to a latent representation z, which a trajectory generation network converts into the final trajectory Y.This describes the typical end-to-end model structure used as the formulation's basis.
- Mixture of Experts: The MoE design combines shared experts and domain-specific experts, with a gated network assigning each expert a weight for input x.The expert functions operate over model representations, and gating weights lie in [0, 1].
- Mixture of Experts: The MoE output aggregates expert computations using their gate-assigned weights.The displayed formulation represents weighted expert contributions for the input.
B. Model Architecture
ARTEMIS fuses multimodal perception with autoregressive waypoint generation and dynamically routed MoE experts. Batch reallocation improves expert processing efficiency, while trajectory refinement incorporates semantic and kinematic information.
- Perception Module: Separate image and point-cloud extractors fuse multimodal features into a unified BEV representation for planning.The perception module follows Transfuser-style fusion with parallel feature extractors and Transformer-based integration.
- Autoregressive Planning Module with Mixture-of-Experts: Autoregressive planning incrementally generates trajectory waypoints while using previous trajectory information and scene-specific expert selection.Each point is sampled from a predicted multimodal distribution, then appended to update the planning-query sequence.
- Autoregressive Planning Module with Mixture-of-Experts: Driving-command imbalance and command–trajectory discrepancies motivate expert selection mechanisms that capture diverse driving behavior without relying solely on commands.The navtrain split contains over 20,000 left-turn, fewer than 10,000 right-turn, and more than 50,000 forward-driving samples.
- Autoregressive Planning Module with Mixture-of-Experts: Batch reallocation groups samples activating the same experts, processes shared and specialized experts, then restores original ordering before producing the next planning query.Sparse routing selects only the top k experts, while sorting and block identification organize expert-specific computation.
- Trajectory Refinement Module: Trajectory refinement combines semantic kinematic optimization with cross-attention to enforce smoothness and kinematic constraints and incorporate scene context.The refinement module uses semantic-map features, trajectory features, agent features, and ego planning features.
3) Trajectory Refinement Module:
The trajectory refinement module optimizes generated points under explicit constraints and then uses cascaded cross-attention to integrate scene and planning context.
- Trajectory Refinement Module: Semantic kinematic optimization generates refined trajectory points using explicit smoothness and kinematic constraints with learnable weights.The optimizer combines semantic-map features with trajectory features before point-by-point decoding.
- Trajectory Refinement Module: Cascaded cross-attention lets trajectory features interact with agent features and ego planning features to further refine the trajectory.The refinement function operates on both agent-query and ego-planning-query features.
4) Training Loss:
ARTEMIS uses staged training: perception and auxiliary tasks are optimized first, followed by end-to-end training of the entire network. The losses combine perception, planning trajectory, and negative log-likelihood objectives.
- Perception is trained first with auxiliary semantic-mapping, classification, and localization tasks to improve training stability.
- The perception-stage loss combines semantic, classification, and bounding-box losses with separate weighting coefficients.Lsem is semantic-map cross-entropy, while Lclass and Lbox are classification and localization losses.
- End-to-end training then optimizes the entire network using planning trajectory loss, negative log-likelihood loss, and perception loss.The overall objective includes Ltraj, LNLL, and Lperception.
A. Dataset
ARTEMIS is trained and tested on NAVSIM, which selects challenging scenarios from OpenScene and provides multimodal driving observations with historical context. The dataset contains 1192 training scenarios and 136 test scenarios.
- NAVSIM selects challenging scenarios from OpenScene while excluding simple scenarios.
- The dataset contains 1192 training scenarios and 136 test scenarios.
- Each sample includes eight camera viewpoints, LiDAR fused from five sensors, map annotations, and 3D object bounding boxes.
B. Evaluation Metrics
NAVSIM evaluates driving performance with PDMS and its extended EPDMS benchmark. The experiments use multimodal camera-LiDAR inputs and an autoregressive configuration that generates and refines trajectory points.
- PDMS provides an intermediate evaluation between open-loop and closed-loop testing and correlates highly with closed-loop metrics.
- PDMS combines No-Collision, Drivable Area Compliance, Time-to-Collision, Comfort, and Ego Vehicle Progress indicators.
- EPDMS extends PDMS with Lane Keeping, Extended Comfort, Driving Direction Compliance, Traffic Light Compliance, and false-alarm filtering.
- The model uses camera and LiDAR inputs, five domain-specific experts, one shared expert, and eight autoregressive planning steps.Each step outputs one trajectory point, followed by refinement into a 4-second trajectory at 2 Hz.
D. Main Results
ARTEMIS achieves strong Navtest performance with a ResNet-34 backbone and state-of-the-art results under extended metrics. Qualitative examples show experts proposing different behaviors that routing fusion incorporates into final trajectories.
- Quantitative Results: 87.0 PDMS is achieved by ARTEMIS with a ResNet-34 backbone on the Navtest split.The paper reports competitive performance across most compared models and superior EP, NC, and C scores.
- Quantitative Results: ARTEMIS achieves state-of-the-art results against same-backbone baselines under Navtest extended metrics.The paper highlights especially strong TTC and EP results.
- Qualitative Results: Fig.5 visualizes trajectories from individual domain-specific experts in different colors across representative Navtest scenarios.
- Qualitative Results: At an intersection, routing fusion prioritizes the straight-ahead trajectory proposed by the orange expert.
- Qualitative Results: At a T-intersection, routing fusion incorporates elements of the yellow expert’s right-turn trajectory while other experts continue straight.
- Qualitative Results: In a roundabout scenario, the red expert correctly captures road features while the other experts incorrectly select a right-front direction.
E. Ablation Study
The ablations show that ARTEMIS depends on autoregressive planning, expert routing, and cascaded refinement, while expert-pool size and refinement depth require calibration. Batch reallocation also substantially improves training throughput.
- Model components: Removing the autoregressive module reduces PDMS by 3.0 points, while removing MoE reduces it by 4.1 points.These results support the roles of temporal waypoint dependencies and dynamically specialized processing.
- Expert routing: The routing network prevents inappropriate expert assignment when explicit driving commands deviate from the actual scene.Single-expert activation also fails to achieve optimal performance, while training shows balanced expert utilization.
- Refinement depth: Increasing refinement depth improves performance through two layers, after which gains stabilize and marginal benefits diminish.The refinement stage also mitigates sampling instabilities during autoregressive generation.
- Expert count: Expanding the expert pool from three to five improves performance, whereas ten experts reduce performance by 1.5 points.The reported explanation is that limited training data disperses resources and creates functional overlap.
- Training speed: Batch reallocation increases training samples per second from 19.2 to 43.5 as batch size grows from 64 to 256.Its overhead is described as negligible relative to expert-network computation under identical hardware conditions.
IV. CONCLUSION
ARTEMIS uses sequential trajectory generation with routed Mixture-of-Experts processing to model evolving trajectories and diverse driving behavior. On NAVSIM, the framework achieves highly competitive performance and is presented as promising for complex-scenario deployment.
- Conclusion: ARTEMIS replaces one-shot trajectory synthesis with sequential decision-making and integrates routed experts to accommodate diverse driving environments.The framework combines autoregressive planning with dedicated routing networks to capture dynamic driving characteristics.