PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging Past and Future: End-to-End Autonomous Driving with Historical Prediction and Planning

Bozhou Zhang, Nan Song, Xin Jin, Li Zhang

arXiv:2503.14182v1cs.ROcs.CV

TL;DR

Existing end-to-end driving methods either omit historical information from motion planning or fail to match its multi-step structure. BridgeAD reformulates motion and planning queries as step-specific multi-step queries and integrates historical information into perception and future planning, achieving state-of-the-art performance on nuScenes in open- and closed-loop evaluations.

  • Problem

    Existing temporal-information methods either use history only in perception or represent historical queries in a way that does not match multi-step motion planning.

  • Method

    BridgeAD represents motion and planning queries by future time step, combining current-frame historical prediction with perception and future-frame historical prediction and planning with motion planning.

  • Results

    BridgeAD achieves state-of-the-art performance on the nuScenes dataset in both open-loop and closed-loop settings.

  • Takeaways & Limitations

    Historical prediction and planning can be integrated across perception, prediction, and planning to improve coherence across future time steps.

  • Takeaways & Limitations

    Closed-loop testing shows that BridgeAD still struggles with safety-critical scenarios and relies heavily on complex post-processing.

Abstract

from arXiv · show

End-to-end autonomous driving unifies tasks in a differentiable framework, enabling planning-oriented optimization and attracting growing attention. Current methods aggregate historical information either through dense historical bird's-eye-view (BEV) features or by querying a sparse memory bank, following paradigms inherited from detection. However, we argue that these paradigms either omit historical information in motion planning or fail to align with its multi-step nature, which requires predicting or planning multiple future time steps. In line with the philosophy of future is a continuation of past, we propose BridgeAD, which reformulates motion and planning queries as multi-step queries to differentiate the queries for each future time step. This design enables the effective use of historical prediction and planning by applying them to the appropriate parts of the end-to-end system based on the time steps, which improves both perception and motion planning. Specifically, historical queries for the current frame are combined with perception, while queries for future frames are integrated with motion planning. In this way, we bridge the gap between past and future by aggregating historical insights at every time step, enhancing the overall coherence and accuracy of the end-to-end autonomous driving pipeline. Extensive experiments on the nuScenes dataset in both open-loop and closed-loop settings demonstrate that BridgeAD achieves state-of-the-art performance.

1. Introduction

End-to-end driving unifies perception, prediction, and planning, but existing temporal-information paradigms either omit motion planning or mismatch its multi-step nature. BridgeAD addresses this gap by using step-specific historical prediction and planning across the pipeline.

  • End-to-end methods unify perception, prediction, and planning to support planning-oriented optimization and improved system coherence.
  • Existing approaches aggregate history through dense BEV features or sparse memory-bank queries inherited from detection.
  • Dense methods use temporal information only in perception, while sparse methods do not align historical queries with multi-step motion planning.
  • BridgeAD decomposes motion and planning queries by future time step, combining current-frame historical prediction with perception and future-frame history with motion planning.
  • BridgeAD represents motion and planning queries as multi-step queries and achieves state-of-the-art performance on nuScenes in open-loop and closed-loop settings.

2. Related work

Related work spans modular perception, motion prediction, and planning methods, alongside increasingly unified end-to-end systems. Recent approaches integrate these tasks while exploring temporal modeling, belief states, vectorized representations, and generative planning.

  • Perception: Perception research covers 3D detection, multi-object tracking, and online mapping using BEV, sparse-query, track-query, and vectorized-map approaches.
  • Motion prediction: Motion prediction methods forecast multi-modal future trajectories using query-centric designs, historical predictions, streaming, and end-to-end image-to-trajectory pipelines.
  • Planning: Planning research includes rule-based, learning-based, belief-state, and end-to-end approaches that connect perception, prediction, and ego-trajectory planning.
  • Planning: Recent unified systems include query-based, vectorized, generative, and sparse-driving frameworks targeting performance and efficiency across autonomous-driving tasks.

3. Methodology

BridgeAD builds an end-to-end pipeline around historical information and multi-step motion and planning queries. Its modules enhance perception, propagate history across future steps, and couple motion prediction with planning.

  • Framework overview: BridgeAD uses an image encoder, history-enhanced perception, and history-enhanced motion planning to process multi-view images into driving outputs.
  • Multi-step query caching: Motion queries are expanded with a future-time-step dimension, while planning queries separately index planning modes and future planning steps.Historical queries from K past frames are stored in a FIFO memory queue.
  • History-enhanced perception: The image encoder extracts multi-view features, which support sparse 3D detection, tracking, and online vectorized mapping.
  • History-enhanced perception: Historical current-frame motion queries interact with object queries through cross-attention before refined perception queries enter later modules.
  • History-enhanced motion planning: Historical motion and planning queries are aggregated into future-frame queries using cross-attention plus step-level and mode-level self-attention.
  • History-enhanced motion planning: Cross-attention aligns historical information with corresponding steps, while self-attention propagates it across all query steps and modes.
  • History-enhanced motion planning: Step-level Mot2Plan interaction couples motion queries with planning queries at corresponding future steps, after which planning trajectories and scores are produced.
  • Training: End-to-end training combines detection, mapping, motion, and planning losses, with regression and classification components for each task.The total loss is L_total = L_det + L_map + L_mot + L_plan.

4. Experiments

BridgeAD is evaluated on nuScenes in open-loop and closed-loop settings across planning, perception, and motion prediction. It achieves state-of-the-art or superior performance, with ablations attributing gains to historical information, multi-step interactions, and self-attention, while maintaining reasonable efficiency.

  • Open-loop planning: BridgeAD achieves state-of-the-art open-loop planning performance against recent end-to-end and world-model methods without using ego status as input.It outperforms methods that rely on ego status despite avoiding this input.
  • Closed-loop planning: A 65% higher NeuroNCAP score than SparseDrive and a 12.4% lower collision rate than UniAD demonstrate stronger closed-loop planning without post-processing.The evaluation uses the NeuroNCAP simulator on nuScenes-based safety-critical scenarios.
  • Perception and motion prediction: BridgeAD achieves superior performance across all motion-prediction metrics and improves both detection and tracking compared with other methods.The reported gains use historical information and multi-step motion-query representations.
  • Ablation study: Removing either History-Enhanced Planning or Step-Level Mot2Plan Interaction significantly reduces planning performance.The ablation identifies historical planning information and surrounding-agent prediction as crucial to ego-vehicle planning.
  • Ablation study: Removing History-Enhanced Motion Prediction lowers motion-prediction performance, while removing Historical Mot2Det Fusion lowers detection and tracking performance.The results indicate that the two modules contribute to different stages of perception and prediction.
  • Ablation study: Without step-level or mode-level self-attention, planning performance significantly decreases because historical information is not propagated across all planning steps and modes.The mechanisms improve planning accuracy and consistency at each time step.
  • Ablation study: The best historical-information aggregation uses 6 time steps for motion queries and 3 for planning queries.The study varies one query type while fixing the other at its selected setting.
  • Efficiency: BridgeAD has 157.2 ms inference latency, compared with 224.3 ms for VAD and 555.6 ms for UniAD.FPS comparisons use a single RTX 3090 with batch size 1 for most models, while UniAD uses an official A100 measurement.

5. Conclusion

BridgeAD integrates historical prediction and planning across the end-to-end autonomous-driving pipeline through multi-step queries, improving coherence across future time steps. Experiments on nuScenes in open-loop and closed-loop settings show superior performance.

  • Conclusion: BridgeAD integrates historical prediction and planning across perception, prediction, and planning stages.Its multi-step queries distinguish future time steps and enable step-specific interactions.
  • Conclusion: Multi-step queries leverage temporal information to improve coherence across future time steps.
  • Conclusion: BridgeAD achieves superior performance on nuScenes in both open-loop and closed-loop scenarios.

6. Methodology

BridgeAD uses sparse perception processing, task-specific regression and classification losses, and a combined objective spanning detection, mapping, motion prediction, and planning.

  • Perception: The perception component uses sparse processing with decoder layers that attend across object queries and aggregate visual features deformably.The Historical Mot2Det Fusion Module then refines object queries and detection outputs using historical information.
  • Loss function: Each task loss is divided into regression and classification components.
  • Loss function: The total loss combines detection, online mapping, motion prediction, and planning losses.Each task loss is weighted by separate regression and classification coefficients.
  • Notation: The paper provides a notation lookup table for the symbols used throughout the method.

7. Experiments

Experiments evaluate perception, prediction, planning, robustness, generalization, and failure cases across nuScenes, NeuroNCAP, and CARLA Bench2Drive settings. The reported results include strong benchmark performance, while ablations and qualitative analyses examine design effects and safety limitations.

  • Open-loop evaluation: Open-loop evaluation covers detection, tracking, mapping, motion prediction, and planning using established task-specific metrics.Metrics include mAP, NDS, AMOTA, AMOTP, IDS, map AP, ADE, FDE, MR, EPA, L2 displacement error, and collision rate.
  • Closed-loop evaluation: A NeuroNCAP score awards full credit for avoiding collisions and partial credit for reducing impact velocity.The score is 5.0 without collision and otherwise decreases according to relative impact speed.
  • Online mapping: The online mapping results are compared with state-of-the-art online mapping and end-to-end methods on the nuScenes validation dataset.
  • Motion forecasting: The model outperforms UniAD and ViP3D on motion-forecasting metrics focused on moving agents.
  • Generalization: On CARLA v2 with Bench2Drive, the method outperforms UniAD and VAD in both open-loop and closed-loop evaluations.The reported metrics include Driving Score and Success Rate.
  • Ablations: Step-level and mode-level self-attention propagate historical information across prediction steps and modes, improving motion-prediction accuracy.
  • Ablations: K = 3 historical frames provide the best balance between efficiency and performance.
  • Failure cases: Failure cases include trajectories leaving the road at curbs and crashes caused by insufficient steering and deceleration.The suggested remedies are road constraints or post-processing, and additional deceleration-focused training or slowing-down enforcement.

8. Limitations and future work

Closed-loop testing indicates that BridgeAD still struggles with safety-critical scenarios and depends heavily on complex post-processing. The paper identifies additional safety-focused training, reinforcement learning, and rule-based planning as directions for future work.

  • Limitations: BridgeAD still struggles with safety-critical scenarios and relies heavily on complex post-processing.The authors describe this as a limitation shared by existing end-to-end methods.
  • Future work: Future work may add more training data for such situations or integrate end-to-end learning with reinforcement learning or rule-based planning.The paper states that current historical planning aggregation remains insufficient despite mitigating some scenarios.

9. Discussion

BridgeAD represents each future time step with distinct motion and planning queries, then groups same-step queries for interaction. This design connects historical information with perception, prediction, and planning as belief-state encoding.

  • Multi-step queries: BridgeAD uses multiple queries for one trajectory, distinguishing each future time step instead of representing the entire trajectory with a single query.For a 3-second trajectory planned at 2 Hz, six queries represent one planning trajectory.
  • Query interaction: Queries corresponding to the same time step interact through designed modules for both surrounding-agent motion and ego-agent planning.
  • Belief-state interpretation: BridgeAD can be viewed as encoding belief states through historical prediction and planning across perception, prediction, and planning.
  • End-to-end scope: Unlike approaches focused on historical predictions for motion forecasting, BridgeAD incorporates historical prediction and planning to optimize the entire end-to-end autonomous-driving pipeline.
Loading 2503.14182v1…