Source-linked AI summary
Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving
Linhan Wang, Zichong Yang, Chen Bai, Guoxiang Zhang, Xiaotong Liu, Xiaoyin Zheng, Xiao-Xiao Long, Chang-Tien Lu, Cheng Lu
TL;DR
End-to-end driving needs better planning representations and richer supervision because video world-model pretraining has yielded limited gains and scenes usually contain only one human trajectory. Drive-JEPA combines V-JEPA driving-video pretraining with simulator-guided multimodal trajectory distillation and momentum-aware selection. It reports state-of-the-art NAVSIM performance, including 87.8 EPDMS on v2 and a 3 PDMS perception-free gain over prior work.
Problem
Video world-model pretraining has brought limited improvements, while single human trajectories inadequately represent driving’s multimodal futures.
Method
Drive-JEPA pretrains a ViT encoder with V-JEPA on driving videos and distills simulator-generated multimodal trajectories into a proposal-centric planner with momentum-aware selection.
Results
Drive-JEPA achieves state-of-the-art results on NAVSIM v1 and v2, while V-JEPA with a simple decoder outperforms prior work by 3 PDMS in perception-free driving.
Takeaways & Limitations
Planning-aligned video representations and diverse simulator-guided supervision support strong perception-free and multimodal end-to-end driving performance.
Takeaways & Limitations
Naive human-trajectory guidance limits proposal multimodality because driving scenes often admit multiple valid choices beyond the single human trajectory.
Abstract
from arXiv · showhide
End-to-end autonomous driving increasingly leverages self-supervised video pretraining to learn transferable planning representations. However, pretraining video world models for scene understanding has so far brought only limited improvements. This limitation is compounded by the inherent ambiguity of driving: each scene typically provides only a single human trajectory, making it difficult to learn multimodal behaviors. In this work, we propose Drive-JEPA, a framework that integrates Video Joint-Embedding Predictive Architecture (V-JEPA) with multimodal trajectory distillation for end-to-end driving. First, we adapt V-JEPA for end-to-end driving, pretraining a ViT encoder on large-scale driving videos to produce predictive representations aligned with trajectory planning. Second, we introduce a proposal-centric planner that distills diverse simulator-generated trajectories alongside human trajectories, with a momentum-aware selection mechanism to promote stable and safe behavior. When evaluated on NAVSIM, the V-JEPA representation combined with a simple transformer-based decoder outperforms prior methods by 3 PDMS in the perception-free setting. The complete Drive-JEPA framework achieves 93.3 PDMS on v1 and 87.8 EPDMS on v2, setting a new state-of-the-art.
1 Introduction
Drive-JEPA addresses limited gains from video pretraining and the single-trajectory supervision bottleneck by combining V-JEPA driving-video pretraining with multimodal trajectory distillation. Its components target planning-aligned representations, diverse trajectory supervision, and stable selection, achieving state-of-the-art NAVSIM results.
- Motivation: Video world-model pretraining has produced only limited improvements for planning, while each driving scene typically supplies one human trajectory despite multimodal futures.These constraints motivate learning representations aligned with planning and obtaining additional trajectory supervision.
- Approach: Drive-JEPA adapts V-JEPA to driving videos to learn predictive, planning-aligned representations from large-scale raw video.The framework uses a ViT-based encoder pretrained with V-JEPA.
- Approach: Multimodal trajectory distillation transfers simulator knowledge into a proposal-centric planner, providing diverse supervision beyond single human trajectories.The method generates waypoint-anchored proposals and selects high-quality multimodal trajectories from a trajectory vocabulary.
- Results: 93.7 PDMS on NAVSIM v1 and 87.8 EPDMS on NAVSIM v2 establish new state-of-the-art results, while perception-free V-JEPA planning exceeds prior work by 3 PDMS.The reported evaluation covers NAVSIM v1, NAVSIM v2, and Bench2Drive.
- Approach: Momentum-aware trajectory selection is designed to enhance driving comfort and stabilize behavior among candidate trajectories.The selection module is one of the framework’s three main components.
2 Related Work
Related work covers end-to-end driving, video world models, and multimodal trajectory planning. Prior methods use direct sensor-to-action learning, learned scene dynamics, or discrete trajectory vocabularies, each motivating the paper’s design choices.
- End-to-end Autonomous Driving: End-to-end driving maps raw sensor observations directly to control actions, avoiding hand-designed intermediate representations in modular pipelines.Prior systems include ALVINN and PilotNet, while later methods add intermediate representations or auxiliary supervision for robustness.
- Video World Models: Video world models predict scene evolution under ego actions, and prior work transfers their learned dynamics or generation capabilities to end-to-end driving.Examples include VaVIM’s causal video model and Epona’s hybrid diffusion–autoregressive predictor.
- Multimodal Trajectory Planning: Discrete multimodal planners cluster continuous actions into fixed trajectory vocabularies and select among them using predicted scores.VADv2 and Hydra-MDP use large fixed vocabularies to represent diverse driving behaviors.
3 Method
Drive-JEPA combines V-JEPA driving-video pretraining with a proposal-centric planner that distills multimodal simulator trajectories and selects trajectories with cross-frame comfort awareness.
- Driving Video Pretraining: V-JEPA pretrains a ViT encoder on large-scale driving videos to learn predictive representations for planning.The objective predicts masked latent representations and uses stop-gradient with an exponential moving-average target encoder to prevent collapse.
- Driving Video Pretraining: The perception-free decoder uses pretrained spatiotemporal features and learnable waypoint queries to predict future BEV positions and headings.A transformer decoder attends to encoded features, and the predicted trajectory is trained with mean squared error against the human trajectory.
- Driving Video Pretraining: V-JEPA pretraining with a simple transformer decoder significantly outperforms prior perception-free planners.The reported result highlights the effectiveness of V-JEPA-based driving-video pretraining for planning.
- Waypoint-anchored Proposals Generation: Waypoint-anchored proposal generation initializes multiple trajectory queries from ego status and positional embeddings, then iteratively refines them using waypoint-based BEV feature sampling.The final proposals contain waypoint positions and headings, with deformable attention aggregating visual features around predicted waypoints.
- Multimodal Trajectories Distillation: Multimodal trajectory distillation uses simulator-scored trajectory centers as pseudo-teachers, replacing single-trajectory guidance to diversify proposals.A vocabulary of 8192 clustered trajectory centers is scored with rule-based simulation, and selected multimodal trajectories guide proposal learning.
- Momentum-aware Trajectory Selection: Momentum-aware trajectory selection scores proposals using both learned proposal quality and comfort relative to the trajectory selected at the previous frame.The comfort term addresses temporal inconsistency that can arise when multimodal distillation increases proposal diversity.
4 Experiments
Drive-JEPA is evaluated on NAVSIM v1, NAVSIM v2, and Bench2Drive using simulation-based and closed-loop driving metrics. It achieves strong benchmark performance, while ablations show complementary effects from video pretraining, multimodal distillation, and momentum-aware selection.
- Evaluation Setup: Drive-JEPA is evaluated on NAVSIM v1, NAVSIM v2, and Bench2Drive, covering open-loop simulation-based and closed-loop interactive driving assessment.NAVSIM reports rule-based simulation metrics, while Bench2Drive evaluates interactive urban routes in CARLA.
- NAVSIM Results: Drive-JEPA surpasses prior methods on NAVSIM v2 while maintaining strong safety, traffic-rule compliance, Ego Progress, and Extended Comfort performance.NAVSIM v2 extends evaluation with richer compliance and comfort metrics.
- Bench2Drive Results: Drive-JEPA achieves the best Driving Score on Bench2Drive and surpasses iPad by 4 in Driving Score.The method also reports very competitive Efficiency.
- Ablation Studies: Multimodal trajectory distillation improves diversity and overall metrics but worsens EC, while momentum-aware selection raises EC to 84.8 and sets a new best EPDMS.Using pseudo-teacher trajectories consistently outperforms using none.
- Ablation Studies: V-JEPA pretraining outperforms mainstream alternatives, and driving-video pretraining further surpasses Epona by 3 PDMS.MAE and DepthAnything could not converge in the reported comparison.
5 Conclusion
The conclusion presents Drive-JEPA as an end-to-end driving framework combining V-JEPA video pretraining with multimodal trajectory distillation. Its reported results are state of the art on NAVSIM and improved in closed-loop Bench2Drive evaluation.
- Conclusion: Drive-JEPA combines V-JEPA video pretraining with multimodal trajectory distillation to mitigate imitation-learning modal collapse.The framework also uses momentum-aware selection to improve temporal stability and comfort.
- Conclusion: Pretraining a ViT encoder on large-scale driving videos yields strong planning representations that support competitive perception-free performance with a simple decoder.The conclusion attributes proposal diversity to simulator-guided pseudo-teacher trajectories.
- Conclusion: Drive-JEPA achieves state-of-the-art results on NAVSIM v1/v2 and improves closed-loop performance on Bench2Drive.
A.1 Extended Predictive Driver Model Score (EPDMS)
EPDMS extends NAVSIM’s PDMS metric with additional rule-compliance and comfort terms. Its multiplicative and additive components jointly score safety, drivable-area adherence, progress, collision timing, lane keeping, and smoothness.
- Metric Definition: EPDMS extends PDMS by adding Driving Direction Compliance, Traffic Light Compliance, Lane Keeping, History Comfort, and Extended Comfort.
- Metric Definition: EPDMS = NC × DAC × DDC × TLC × 5 × (EP + TTC) + 2 × (LK + HC + EC).
- Metric Components: The metric includes No at-fault Collision, Drivable Area Compliance, Ego Progress, and Time to Collision alongside the newly introduced terms.
- Rule Compliance: DDC measures legal travel direction, TLC measures obedience to traffic-light phases, and LK measures staying near the current lane centerline.
- Comfort: HC evaluates comfort after prepending 1.5 seconds of the human driver’s recent motion, while EC checks smoothness across consecutive time steps.
A.2 Diversity (D)
The diversity metric is computed from rasterized trajectory occupancy masks. Each trajectory is represented as a 2D corridor formed by buffering its polyline before grid projection.
- Diversity Computation: Diversity uses trajectory IoU computed from 2D occupancy masks created by buffering each trajectory polyline with a 2-meter width.The buffered corridor is projected onto a grid for comparison.
A.3 Metrics used in Bench2Drive
Bench2Drive reports route success, driving performance, efficiency, and comfort through distinct metrics covering completion, infractions, speed relative to traffic, and vehicle smoothness.
- Success Rate measures the proportion of routes completed within the allotted time without traffic violations.
- Driving Score combines route completion with penalties for infractions under the official CARLA metric.
- Efficiency checks for excessively low ego-vehicle speed relative to nearby traffic.
- Comfort evaluates longitudinal and lateral accelerations, yaw dynamics, longitudinal jerk, and maximum jerk-vector magnitude.
B More details
The additional details specify trajectory-selection sampling, input-resolution comparisons, and the table documenting image-resolution settings.
- A simulated EPDMS threshold of 0.95 selects high-quality trajectories from the vocabulary.When more than Npseudo trajectories exceed the threshold, training uniformly samples Npseudo trajectories from that subset.
- Table 8 documents input image resolution by method.
- The proposed setting uses only the front camera at 512 × 256 resolution, with It and It−1 stacked into a 2 × 512 × 256 tensor.
C More visualization
The visualization shows that multimodal trajectory distillation prevents proposal collapse and produces a multimodal proposal distribution.
- Without Multimodal Trajectory Distillation, trajectory proposals collapse into one mode.
- With Multimodal Trajectory Distillation, trajectory proposals exhibit a multimodal distribution.
NeurIPS Paper Checklist
The checklist reports that the paper’s empirical contributions and experimental reproducibility are documented, while limitations and code access were incomplete at submission.
- The checklist confirms that the stated contributions and reported 93.3 PDMS on NAVSIM v1 and 87.8 EPDMS on v2 are supported by experiments.
- The paper does not include a dedicated limitations discussion, although it plans to cover the single-front-camera assumption, simulator-generated pseudo-teachers, and computational requirements.
- The checklist identifies scope and robustness considerations when claims are tested on only a few datasets or runs.
- The paper has no formal theoretical results or proofs because its contributions are empirical.
- The checklist judges experimental-result reproducibility adequate based on architectural, objective, module, and experimental descriptions.
- All datasets are publicly released, but anonymized code was unavailable at submission and was planned for release upon acceptance.