AD-E2E-JEPA: A Joint-Embedding Predictive Architecture For End-to-End Autonomous Driving

• Existing E2EAD relies heavily on imitation of human trajectories, which can be suboptimal and noisy, while world models can predict futures under candidate actions for goal-directed planning.Source: §1 · p. 2
• Without training a driving policy, AD-E2E-JEPA selects candidate trajectories for a goal specified by an image, using human trajectories only as action conditioning.Source: §1 · p. 2
• The paper adapts JEPA to E2EAD for goal-conditioned zero-shot planning and reports planning efficiency, geodesic accuracy, and reliability alongside success rate.Source: §1 · p. 2
• Existing JEPA world models trade planning accuracy against efficiency: efficient models can be inaccurate, while accurate models can require over a minute per scene.Source: §1 · p. 2
• AD-E2E-JEPA uses a learnable, SIGReg-regularized projector to reduce patch embeddings and planning latency while preserving planning performance.The paper presents this design as a response to the efficiency–performance trade-off and notes potential application beyond E2EAD.Source: §1 · p. 2
• 23.7 EPDMS: AD-E2E-JEPA improves over LeWM across 12,146 NAVSIM test scenes while requiring 100× less planning time than DINO-WM/JEPA-WM.With 8,192 candidate trajectories, its best variant reaches 72.9 EPDMS, with 2.8-meter average displacement error and 2.0-degree mean absolute heading error.Source: §1 · p. 2
• 85.4 EPDMS versus 80.2: the self-supervised pretrained projector improves downstream imitation learning in a simple ViT-based E2E architecture.Source: §1 · p. 2

2.1 WORLD MODELS

• World models span pixel-space prediction, latent dynamics, and latent-only prediction; whether visually realistic generative models reliably capture physical laws for planning remains unclear.Source: §2.1 · p. 3
• JEPA predicts target representations from context and uses regularization to avoid collapse; action-conditioned latent predictions have supported zero-shot planning across modalities.Related planning directions include temporal straightening, lower-dimensional projection, hierarchical planning, sparse representations, and adaptive updates.Source: §2.1 · p. 3

2.2 END-TO-END AUTONOMOUS DRIVING

• E2EAD maps raw sensor inputs to planning actions, but common imitation and scoring-based approaches still depend on human-policy imitation or dense trajectory-level supervision.Source: §2.2 · p. 3
• Prior JEPA driving work covers perception, representation learning, world modeling, and scoring-based planning; this paper instead investigates zero-shot planning without task-specific imitation learning.Source: §2.2 · p. 3

3 METHOD

• AD-E2E-JEPA uses a shared learnable projector and SIGReg to compress patch embeddings while preserving informative representations for planning.The DINOv3 encoder is frozen, and the projector is shared between context-history and target-future branches.Source: Figure 2 · p. 4, §3.2.2 · p. 5

3.1 PRELIMINARY: TASK FORMULATION OF E2EAD

• The driving task uses front-camera images and historical ego poses to predict future ego poses that safely make progress.Each pose contains planar position and heading, expressed relative to the current ego pose.Source: §3.1 · p. 4

3.2.1 JEPA-WM ADAPTATION FOR E2EAD

• The JEPA-WM adaptation uses a frozen DINOv3 encoder and an AdaLN predictor with RoPE to predict future embeddings from history and actions.The action is the relative pose change between consecutive frames, and the predictions are supervised with MSE against frozen-encoder targets.Source: §3.2.1 · pp. 4–5

3.2.2 AD-E2E-JEPA

• 16× fewer spatial patches and 4× lower embedding dimensionality result from the projector, which reduces ViT-L embeddings from 1024 to 256 dimensions.SIGReg encourages projected embeddings to follow an isotropic Gaussian distribution and helps prevent representation collapse; optional rollout training adds future-horizon consistency losses.Source: §3.2.2 · pp. 5–6

3.3 ZERO-SHOT GOAL-CONDITIONED PLANNING

• Zero-shot goal-conditioned planning evaluates world models against a future observation as the goal, without training a driving policy.The paper also reports driving performance, planning efficiency, geodesic accuracy, and reliability metrics.Source: §3.3 · p. 6, §3.3.2 · p. 6

3.3.1 PLANNING WITH AD-E2E-JEPA

• Planning selects the candidate trajectory whose predicted future latent embedding is closest to the latent embedding of the future-image goal.Candidates are drawn from a clustered driving-trajectory vocabulary subsampled at evenly spaced angular intervals to balance efficiency and trajectory granularity.Source: §3.3.1 · p. 6

3.3.2 METRICS

• Evaluation reports EPDMS, EPDMS†, planning time, final-pose displacement errors, and Top-1 and Top-5 hit rates.EPDMS includes multiplicative safety terms, whereas EPDMS† reports its weighted component without those terms; hit rate measures whether the ground-truth trajectory ranks among the lowest-cost candidates.Source: §3.3.2 · pp. 6–7

3.4 DOWNSTREAM TRANSFER FOR IMITATION LEARNING

• For imitation learning, the pretrained encoder and projector are retained, the patch predictor is discarded, and a trajectory decoder is fine-tuned.The comparison is against variants with a randomly initialized projector.Source: §3.4 · p. 7

4.1 DATASETS

• Training used NAVSIM navtrain’s 10 hours of video sampled at 2 Hz for fair comparison, while AD-E2E-JEPA was also scaled to 70 hours from trainval.Source: §4.1 · p. 7

4.2 IMPLEMENTATION DETAILS

• The experiments used four context frames and eight future frames, 30 training epochs, and a default SIGReg weight of 0.09.The rollout-loss variant uses all eight future frames; the default variant uses only the first future frame.Source: §4.2 · p. 7

4.3 ZERO-SHOT PLANNING PERFORMANCE

• 72.9 EPDMS and 86.5 EPDMS† with 2.8 m FDE were achieved by the 8,192-candidate variant on the full 12,146-scene test set.At 256 candidates, the trainval model with rollout loss achieved 67.3 EPDMS, 84.1 EPDMS†, and 4.0 m FDE.Source: Table 2 · p. 8, §4.3 · p. 9
• DINO-WM and JEPA-WM scored 68.3 and 74.2 EPDMS on 100 scenes but required 91.8 and 101.0 seconds per scene, versus 0.8 seconds for AD-E2E-JEPA.LeWM was efficient but had lower driving performance, geodesic accuracy, and hit rates; Figure 3 depicts its shifted selected trajectory and the stronger trajectory matches of the other models.Source: §4.3 · pp. 8–9, Figure 3 · p. 8
• On 100 scenes, rollout loss and trainval training produced 3.2 m FDE and 47%/78% top-1/top-5 hit rates, comparable to or better than DINO-WM and JEPA-WM on those metrics.The model’s EPDMS was lower on this subset; the text notes that 100-scene variance may partly explain it and that the gap narrows on the full test set.Source: §4.3 · p. 9

4.4 TRANSFER LEARNING PERFORMANCE

• Projector transfer was studied without additional self-supervised encoder pretraining, with other NAVSIMv2 methods included as references rather than direct baselines.The table caption identifies view type, perception dependence, and input-frame count as benchmark descriptors.Source: §4.4 · p. 9, Table 3 · p. 9

• AD-E2E-JEPA evaluates world-model planning separately from driving-policy learning through zero-shot goal-conditioned planning.Source: §5 · p. 9
• A learnable projector with SIGReg compresses latent representations for planning while preserving their information content.Source: §5 · pp. 9–10
• 72.9 EPDMS, 2.8 m displacement error, and 2.0-degree mean heading error were achieved by the best variant using 8,192 candidate trajectories without policy training.Source: §5 · p. 10

• SIGReg uses the default settings also used in LeWorld, with the integral approximated using 17 knots over [0, 3].Source: §A.1 · p. 13
• The configuration uses 1,024 random projection directions.Source: §A.1 · p. 13
• T(·) denotes the univariate Epps–Pulley test used in the SIGReg configuration.Source: §A.1 · p. 13

• The rollout implementation adapts Terver et al. (2025) to the full future horizon of F frames.Source: §A.2 · p. 13
• Each prediction window contains K + 2 consecutive frames, using the first K + 1 as input to predict the corresponding one-step-shifted frames.Source: §A.2 · p. 13
• In NAVSIM, the rollout setting uses F = 8 future frames and K + 1 = 4 input frames.Source: §A.2 · p. 13

A.2.1 TEACHER-FORCING LOSS

• Teacher-forcing prediction uses ground-truth context embeddings as input, without autoregressive rollout.Source: §A.2.1 · p. 14
• At k = 0, the loss supervises the entire prediction window.Source: §A.2.1 · p. 14
• For subsequent teacher-forced predictions (k ≥ 1), supervision covers only the final time step of the prediction window.Source: §A.2.1 · p. 14
• Each prediction window is generated from context embeddings and action embeddings.Source: §A.2.1 · p. 14

A.2.2 ROLLOUT CONSISTENCY LOSS

• Rollout consistency performs one F-step autoregressive rollout from the initial context window, maintaining K + 1 embeddings as context.Source: §A.2.2 · p. 14
• The initial teacher-forcing prediction is reused as the first autoregressive prediction, without stop-gradient through its initial context.Source: §A.2.2 · p. 14
• TBPTT stops gradients through the autoregressive context before subsequent predictor calls, while each horizon loss remains differentiable through its current call.Source: §A.2.2 · p. 14
• After each prediction, the context drops its earliest embedding and appends the final predicted embedding.Source: §A.2.2 · p. 14
• For horizons k = 2, …, F, each consistency loss supervises only the final prediction; the first step is excluded because teacher-forcing already supervises it.Source: §A.2.2 · p. 14

A.2.3 SIGREG

• SIGReg is applied across all time steps from t − K through t + F, and the rollout setting uses the overall loss in Eq. (12).Source: §A.2.3 · p. 15

A.3 METRICS

• EPDMS combines multiplicative safety terms with a weighted-average driving-quality term, while EPDMS† reports the quality component without the safety terms.Source: §A.3 · p. 15
• Planning time is averaged per scene on one NVIDIA A100 80 GB GPU, and FDE and Δx, Δy, and Δθ quantify final-pose position and heading errors.Source: §A.3 · p. 15
• Top-1 and Top-5 hit rates measure how often the ground-truth trajectory ranks among the lowest-cost candidates in the top one or top five.Source: §A.3 · p. 15

A.4 EPDMS DETAILS FOR ZERO-SHOT GOAL-CONDITIONED PLANNING

• EPDMS details are reported for 100 subsampled test scenes in Table 4 and all 12,146 test scenes in Table 5.Source: §A.4 · p. 16, Table 4 · p. 16, Table 5 · p. 16

A.5 DOWNSTREAM IMITATION LEARNING

• The downstream imitation-learning model uses a pretrained DINOv3 encoder and the proposed patch projector, then predicts future trajectories from projected patch embeddings and a trajectory query.Source: §A.5 · pp. 16–17